You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/05/29 00:20:36 UTC
[GitHub] [incubator-pinot] amrishlal opened a new pull request #6998: [WIP] Support json path expressions in query.
amrishlal opened a new pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998
## Description
Work in progress for adding support for using json path expressions in queries.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
* [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
* [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
## Release Notes
<!-- If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release. -->
<!-- If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text.
-->
## Documentation
<!-- If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r649350918
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
Review comment:
Better yet, compile the hard-coded translated query and assert that they generate the same expression list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648409105
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
Review comment:
Do you mind writing the expected converted query as comments for better visualization ? This is applicable for all tests. It is just one additional line of comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648415813
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
Review comment:
Why should the alias be `min(minus(jsonColum.id, '5'))` and not `MIN(jsonColumn.id - 5)` ?
Also, have we tested that using scalar transformation function (MINUS) inside aggregation function (MIN) works correctly ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d7c4de8) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/929925568ce01b0c04d129253668a33e14ca1615?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9299255) will **increase** coverage by `0.10%`.
> The diff coverage is `74.86%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
+ Coverage 73.33% 73.43% +0.10%
Complexity 12 12
============================================
Files 1453 1454 +1
Lines 72013 72194 +181
Branches 10428 10464 +36
============================================
+ Hits 52809 53016 +207
+ Misses 15667 15640 -27
- Partials 3537 3538 +1
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `42.15% <38.25%> (+0.15%)` | :arrow_up: |
| unittests | `65.43% <74.31%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.30% <69.56%> (-0.92%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `74.67% <74.67%> (ø)` | |
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `90.54% <100.00%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [.../helix/core/minion/MinionInstancesCleanupTask.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9NaW5pb25JbnN0YW5jZXNDbGVhbnVwVGFzay5qYXZh) | `57.14% <0.00%> (-14.29%)` | :arrow_down: |
| [...elix/core/periodictask/ControllerPeriodicTask.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3BlcmlvZGljdGFzay9Db250cm9sbGVyUGVyaW9kaWNUYXNrLmphdmE=) | `80.00% <0.00%> (-6.67%)` | :arrow_down: |
| [...mpl/dictionary/DoubleOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT2ZmSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `57.44% <0.00%> (-5.32%)` | :arrow_down: |
| [...core/query/executor/ServerQueryExecutorV1Impl.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=) | `80.54% <0.00%> (-4.87%)` | :arrow_down: |
| [...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `72.34% <0.00%> (-4.26%)` | :arrow_down: |
| [...pinot/core/query/request/context/TimerContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvVGltZXJDb250ZXh0LmphdmE=) | `91.66% <0.00%> (-4.17%)` | :arrow_down: |
| ... and [29 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [9299255...d7c4de8](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648422408
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
Review comment:
Especially for the case where path expression appears as an argument inside an aggregation function
MIN(jsonColumn.id - 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657510243
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
Review comment:
Looking into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648409105
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
Review comment:
Do you mind writing the expected converted query as comments for better visualization. This is applicable for all tests ? It is just one additional line of comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648413653
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
Review comment:
For all the query rewrite tests written here, we should also write end to end query execution tests to make sure results are correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-867230470
> Please also update the PR description queries where the column is missing in `json_match`
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657510011
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(
+ getJsonPath(parts, true) + getOperatorSQL(kind) + getLiteralSQL(right.getLiteral(), false))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ left.clear();
+ left.setType(ExpressionType.FUNCTION);
+ left.setFunctionCall(getJsonExtractFunction(parts, getColumnTypeForLiteral(right.getLiteral())));
+ }
+ }
+ }
+ break;
+ }
+ case IS_NULL:
+ case IS_NOT_NULL: {
+ Expression operand = operands.get(0);
+ if (operand.getType() == ExpressionType.IDENTIFIER) {
+ String[] parts = getIdentifierParts(operand.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(
+ new StringLiteralAstNode(getJsonPath(parts, true) + getOperatorSQL(kind))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ operand.clear();
+ operand.setType(ExpressionType.FUNCTION);
+ operand.setFunctionCall(getJsonExtractFunction(parts, DataSchema.ColumnDataType.JSON));
+ }
+ }
+ }
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * @return A string array containing all the parts of an identifier. An identifier may have one or more parts that
+ * are joined together using <DOT>. For example the identifier "testTable.jsonColumn.name.first" consists up of
+ * "testTable" (name of table), "jsonColumn" (name of column), "name" (json path), and "first" (json path). The last
+ * two parts when joined together (name.first) represent a JSON path expression.
+ */
+ private static String[] getIdentifierParts(Identifier identifier) {
+ return identifier.getName().split("\\.");
Review comment:
That won't work, because the second argument is a regex pattern string and hence it will take '.' to mean any character.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-867145737
Please also update the PR description queries where the column is missing in `json_match`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648413653
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
Review comment:
For all the query rewrite tests written here, we should also write end to end query execution tests to make sure rewritten queries work properlu
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
Review comment:
For all the query rewrite tests written here, we should also write end to end query execution tests to make sure rewritten queries work properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9bc4c3) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/02d54b9d986a72e6155014b6e0245d10d85ff6c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (02d54b9) will **decrease** coverage by `7.69%`.
> The diff coverage is `74.45%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.17% 65.47% -7.70%
Complexity 12 12
============================================
Files 1451 1452 +1
Lines 71824 72004 +180
Branches 10412 10445 +33
============================================
- Hits 52554 47142 -5412
- Misses 15738 21455 +5717
+ Partials 3532 3407 -125
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.47% <74.45%> (+0.06%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `82.43% <0.00%> (-8.11%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.55% <70.83%> (-0.93%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `74.67% <74.67%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [345 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [02d54b9...d9bc4c3](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657540102
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r656758757
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -163,7 +163,7 @@ protected void processSegments(int taskIndex) {
LOGGER.error("Caught exception while executing operator of index: {} (query: {})", operatorIndex, _queryContext,
e);
_blockingQueue.offer(new IntermediateResultsBlock(e));
- return;
+ throw e;
Review comment:
Reverting this change as I am no longer able to reproduce the case. But if I recall correctly, I was seeing exceptions thrown in JSON_EXTRACT_SCALAR getting eaten up and not showing up in QueryConsole while running under QuickStart. If it reappears it should be easy to catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657510093
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657509397
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/StatementOptimizer.java
##########
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+
+
+/**
+ * Interface for optimizing a particular class of SQL statement. Optimizers that implement this interface may modify
+ * several or all parts of the SQL statement.
+ */
+public interface StatementOptimizer {
+
+ /** Optimize the given SQL statement. */
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema);
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-867230470
> Please also update the PR description queries where the column is missing in `json_match`
Done
> Please also update the PR description queries where the column is missing in json_match
Done
> Please evaluate the overhead of this new optimizer to the existing queries before merging
I have added the following check at the beginning of JsonStatementOptimizer.optimize function:
```
// If schema doesn't have any JSON columns, there is no need to run this optimizer.
if (schema == null || !schema.hasJSONColumn()) {
return;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9bc4c3) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/02d54b9d986a72e6155014b6e0245d10d85ff6c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (02d54b9) will **decrease** coverage by `7.69%`.
> The diff coverage is `74.45%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.17% 65.47% -7.70%
Complexity 12 12
============================================
Files 1451 1452 +1
Lines 71824 72004 +180
Branches 10412 10445 +33
============================================
- Hits 52554 47142 -5412
- Misses 15738 21455 +5717
+ Partials 3532 3407 -125
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.47% <74.45%> (+0.06%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `82.43% <0.00%> (-8.11%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.55% <70.83%> (-0.93%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `74.67% <74.67%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [345 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [02d54b9...d9bc4c3](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648426337
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
+ Assert.assertEquals(pinotQuery.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.id))]))");
+
+ Assert.assertEquals(pinotQuery.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+ }
+
+ /** Test that a json path expression in HAVING clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupByHaving() {
+
+ // Test HAVING clause with a STRING value extracted using json path expression.
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.name.last, count(*) FROM testTable GROUP BY jsonColumn.name.last HAVING jsonColumn.name.last = 'mouse'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.name.last))]))");
+
+ Assert.assertEquals(pinotQuery1.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+
+ Assert.assertEquals(pinotQuery1.getHavingExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:LITERAL, literal:<Literal stringValue:mouse>)]))");
+
+ // Test HAVING clause with a DOUBLE value extract using json path expression.
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.name.last, count(*) FROM testTable GROUP BY jsonColumn.name.last HAVING jsonColumn.name.last = 'mouse'");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.name.last))]))");
+
+ Assert.assertEquals(pinotQuery2.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+
+ Assert.assertEquals(pinotQuery2.getHavingExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:LITERAL, literal:<Literal stringValue:mouse>)]))");
+ }
+
+ /** Test a complex SQL statement with json path expression in SELECT, WHERE, and GROUP BY clauses. */
+ @Test
+ public void testJsonSelectFilterGroupBy() {
+ BrokerRequest sqlBrokerRequest = SQL_COMPILER.compileToBrokerRequest(
+ "SELECT jsonColumn.name.last, count(*) FROM testTable WHERE jsonColumn.id = 101 GROUP BY jsonColumn.name.last");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
+ Assert.assertEquals(pinotQuery.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.name.last))]))");
+
+ Assert.assertEquals(pinotQuery.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+
+ Assert.assertEquals(pinotQuery.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+ }
+
+ /** Test an aggregation function over json path expression in SELECT clause. */
+ @Test
+ public void testStringFunctionOverJsonPathSelectExpression() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT UPPER(jsonColumn.name.first) FROM testTable");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
+ Assert.assertEquals(pinotQuery.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:UPPER, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.first>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))])), Expression(type:IDENTIFIER, identifier:Identifier(name:upper(jsonColumn.name.first)))]))");
+ }
+
+ /** Test a numerical function over json path expression in SELECT clause. */
+ @Test
+ public void testNumericalFunctionOverJsonPathSelectExpression() {
+
+ // Test without user-specified alias.
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT MAX(jsonColumn.id) FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:MAX, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:DOUBLE>), Expression(type:LITERAL, literal:<Literal doubleValue:-Infinity>)]))])), Expression(type:IDENTIFIER, identifier:Identifier(name:max(jsonColumn.id)))]))");
+
+ // Test with user-specified alias.
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT MAX(jsonColumn.id) AS x FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:MAX, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:DOUBLE>), Expression(type:LITERAL, literal:<Literal doubleValue:-Infinity>)]))])), Expression(type:IDENTIFIER, identifier:Identifier(name:x))]))");
+
+ // Test with nested function calls (minus function being used within max function).
+ BrokerRequest sqlBrokerRequest3 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT MAX(jsonColumn.id - 5) FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery3, SCHEMA);
+
Review comment:
Let's cover transform functions as well please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter commented on pull request #6998: [WIP] Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bea9c1c) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/79507da8edff75e931226f31fe349483f55e4406?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (79507da) will **decrease** coverage by `7.76%`.
> The diff coverage is `71.95%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.22% 65.46% -7.77%
Complexity 12 12
============================================
Files 1451 1452 +1
Lines 71824 71987 +163
Branches 10412 10439 +27
============================================
- Hits 52591 47123 -5468
- Misses 15700 21461 +5761
+ Partials 3533 3403 -130
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.46% <71.95%> (+0.05%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `82.43% <0.00%> (-8.11%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `71.51% <71.51%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../core/startree/plan/StarTreeTransformPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlVHJhbnNmb3JtUGxhbk5vZGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [335 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [79507da...bea9c1c](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648409045
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
Review comment:
(nit) please add brief javadocs
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
Review comment:
Do you mind writing the expected converted query as comments for better visualization ? It is just one additional line of comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648415813
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
Review comment:
Why should the alias be `min(minus(jsonColum.id, '5'))` and not `MIN(jsonColumn.id - 5)` ?
Also, have we tested that using scalar transformation function (minus) inside aggregation function (min) works correctly ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648425602
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
Review comment:
Can you add a test where user has specified an alias ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9cb0614) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/709abb01dabe8487856c1c2a2cffb01795f2a00c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (709abb0) will **decrease** coverage by `31.84%`.
> The diff coverage is `14.17%`.
> :exclamation: Current head 9cb0614 differs from pull request most recent head e3bb50d. Consider uploading reports for the commit e3bb50d to get more accurate results
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
=============================================
- Coverage 73.48% 41.64% -31.85%
+ Complexity 91 7 -84
=============================================
Files 1490 1491 +1
Lines 73153 73408 +255
Branches 10525 10573 +48
=============================================
- Hits 53756 30570 -23186
- Misses 15893 40254 +24361
+ Partials 3504 2584 -920
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.64% <14.17%> (-0.01%)` | :arrow_down: |
| unittests | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `70.08% <0.00%> (-22.50%)` | :arrow_down: |
| [...rc/main/java/org/apache/pinot/spi/data/Schema.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9TY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-75.32%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `11.81% <11.81%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `92.30% <71.42%> (-7.70%)` | :arrow_down: |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `66.04% <100.00%> (-5.18%)` | :arrow_down: |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/spi/config/table/TableType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1RhYmxlVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [952 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [709abb0...e3bb50d](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657510243
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
Review comment:
Fixed. The `.` part is already being handled, but I think checking column first would be a bit more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (645c582) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/709abb01dabe8487856c1c2a2cffb01795f2a00c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (709abb0) will **decrease** coverage by `7.93%`.
> The diff coverage is `76.73%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.48% 65.54% -7.94%
Complexity 91 91
============================================
Files 1490 1491 +1
Lines 73153 73393 +240
Branches 10525 10567 +42
============================================
- Hits 53756 48108 -5648
- Misses 15893 21886 +5993
+ Partials 3504 3399 -105
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.54% <76.73%> (+0.07%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `17.32% <0.00%> (-53.91%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.82% <73.91%> (-0.76%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `79.22% <79.22%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [349 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [709abb0...645c582](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657509428
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(
+ getJsonPath(parts, true) + getOperatorSQL(kind) + getLiteralSQL(right.getLiteral(), false))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ left.clear();
+ left.setType(ExpressionType.FUNCTION);
+ left.setFunctionCall(getJsonExtractFunction(parts, getColumnTypeForLiteral(right.getLiteral())));
+ }
+ }
+ }
+ break;
+ }
+ case IS_NULL:
+ case IS_NOT_NULL: {
+ Expression operand = operands.get(0);
+ if (operand.getType() == ExpressionType.IDENTIFIER) {
+ String[] parts = getIdentifierParts(operand.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(
+ new StringLiteralAstNode(getJsonPath(parts, true) + getOperatorSQL(kind))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ operand.clear();
+ operand.setType(ExpressionType.FUNCTION);
+ operand.setFunctionCall(getJsonExtractFunction(parts, DataSchema.ColumnDataType.JSON));
+ }
+ }
+ }
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * @return A string array containing all the parts of an identifier. An identifier may have one or more parts that
+ * are joined together using <DOT>. For example the identifier "testTable.jsonColumn.name.first" consists up of
+ * "testTable" (name of table), "jsonColumn" (name of column), "name" (json path), and "first" (json path). The last
+ * two parts when joined together (name.first) represent a JSON path expression.
+ */
+ private static String[] getIdentifierParts(Identifier identifier) {
+ return identifier.getName().split("\\.");
+ }
+
+ /**
+ * Builds a json path expression when given identifier parts. For example,given [jsonColumn, name, first], this
+ * function will return "$.name.first" as json path expression.
+ * @param parts identifier parts
+ * @param applyDoubleQuote delimit json path with double quotes if true; otherwise, don't delimit json path.
+ * @return JSON path expression associated with the given identifier parts.
+ */
+ private static String getJsonPath(String[] parts, boolean applyDoubleQuote) {
+ StringBuilder builder = new StringBuilder();
+ if (applyDoubleQuote) {
+ builder.append("\"");
+ }
+
+ builder.append("$");
+ for (int i = 1; i < parts.length; i++) {
+ builder.append(".").append(parts[i]);
+ }
+
+ if (applyDoubleQuote) {
+ builder.append("\"");
+ }
+
+ return builder.toString();
+ }
+
+ /** @return true if specified column has column datatype of JSON; otherwise, return false */
+ private static boolean isValidJSONColumn(String columnName, @Nullable Schema schema) {
+ return schema != null && schema.hasColumn(columnName) && schema.getFieldSpecFor(columnName).getDataType()
+ .equals(FieldSpec.DataType.JSON);
+ }
+
+ /** @return true if specified column has a JSON Index. */
+ private static boolean isIndexedJSONColumn(String columnName, @Nullable TableConfig config) {
+ return config != null && config.getIndexingConfig().getJsonIndexColumns().contains(columnName);
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657510745
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(
+ getJsonPath(parts, true) + getOperatorSQL(kind) + getLiteralSQL(right.getLiteral(), false))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ left.clear();
+ left.setType(ExpressionType.FUNCTION);
+ left.setFunctionCall(getJsonExtractFunction(parts, getColumnTypeForLiteral(right.getLiteral())));
+ }
+ }
+ }
+ break;
+ }
+ case IS_NULL:
+ case IS_NOT_NULL: {
+ Expression operand = operands.get(0);
+ if (operand.getType() == ExpressionType.IDENTIFIER) {
+ String[] parts = getIdentifierParts(operand.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(
+ new StringLiteralAstNode(getJsonPath(parts, true) + getOperatorSQL(kind))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ operand.clear();
+ operand.setType(ExpressionType.FUNCTION);
+ operand.setFunctionCall(getJsonExtractFunction(parts, DataSchema.ColumnDataType.JSON));
+ }
+ }
+ }
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * @return A string array containing all the parts of an identifier. An identifier may have one or more parts that
+ * are joined together using <DOT>. For example the identifier "testTable.jsonColumn.name.first" consists up of
+ * "testTable" (name of table), "jsonColumn" (name of column), "name" (json path), and "first" (json path). The last
+ * two parts when joined together (name.first) represent a JSON path expression.
+ */
+ private static String[] getIdentifierParts(Identifier identifier) {
+ return identifier.getName().split("\\.");
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657429450
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/QueryOptimizer.java
##########
@@ -61,16 +67,24 @@ public void optimize(BrokerRequest brokerRequest, @Nullable Schema schema) {
}
}
- /**
- * Optimizes the given SQL query.
- */
+ /** Optimizes the given SQL query. */
public void optimize(PinotQuery pinotQuery, @Nullable Schema schema) {
+ optimize(pinotQuery, null, schema);
+ }
+
+ /** Optimizes the given SQL query. */
+ public void optimize(PinotQuery pinotQuery, @Nullable TableConfig config, @Nullable Schema schema) {
Review comment:
(nit) to be more clear
```suggestion
public void optimize(PinotQuery pinotQuery, @Nullable TableConfig tableConfig, @Nullable Schema schema) {
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/StatementOptimizer.java
##########
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+
+
+/**
+ * Interface for optimizing a particular class of SQL statement. Optimizers that implement this interface may modify
+ * several or all parts of the SQL statement.
+ */
+public interface StatementOptimizer {
+
+ /** Optimize the given SQL statement. */
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema);
Review comment:
(nit) to be more clear
```suggestion
public void optimize(PinotQuery query, @Nullable TableConfig tableConfig, @Nullable Schema schema);
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
Review comment:
Why `Locale.ROOT`? I believe we use local (`toLowerCase()`) in other places, so better to keep it consistent
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
Review comment:
Same here, check whether the identifier is a column before extracting the parts from it
##########
File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
##########
@@ -345,23 +345,29 @@ private BrokerResponseNative handleSQLRequest(long requestId, String query, Json
if (offlineTableName != null && realtimeTableName != null) {
// Hybrid
offlineBrokerRequest = getOfflineBrokerRequest(brokerRequest);
- _queryOptimizer.optimize(offlineBrokerRequest.getPinotQuery(), schema);
+ _queryOptimizer.optimize(offlineBrokerRequest.getPinotQuery(),
+ _tableCache.getTableConfig(TableNameBuilder.OFFLINE.tableNameWithType(rawTableName)), schema);
Review comment:
Same for other places
```suggestion
_queryOptimizer.optimize(offlineBrokerRequest.getPinotQuery(), _tableCache.getTableConfig(offlineTableName), schema);
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
Review comment:
Before extracting the parts from the identifier, we should first check whether the identifier is a column (note that we allow column name with `.` as of now)
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(
+ getJsonPath(parts, true) + getOperatorSQL(kind) + getLiteralSQL(right.getLiteral(), false))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ left.clear();
+ left.setType(ExpressionType.FUNCTION);
+ left.setFunctionCall(getJsonExtractFunction(parts, getColumnTypeForLiteral(right.getLiteral())));
+ }
+ }
+ }
+ break;
+ }
+ case IS_NULL:
+ case IS_NOT_NULL: {
+ Expression operand = operands.get(0);
+ if (operand.getType() == ExpressionType.IDENTIFIER) {
+ String[] parts = getIdentifierParts(operand.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(
+ new StringLiteralAstNode(getJsonPath(parts, true) + getOperatorSQL(kind))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ operand.clear();
+ operand.setType(ExpressionType.FUNCTION);
+ operand.setFunctionCall(getJsonExtractFunction(parts, DataSchema.ColumnDataType.JSON));
+ }
+ }
+ }
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * @return A string array containing all the parts of an identifier. An identifier may have one or more parts that
+ * are joined together using <DOT>. For example the identifier "testTable.jsonColumn.name.first" consists up of
+ * "testTable" (name of table), "jsonColumn" (name of column), "name" (json path), and "first" (json path). The last
+ * two parts when joined together (name.first) represent a JSON path expression.
+ */
+ private static String[] getIdentifierParts(Identifier identifier) {
+ return identifier.getName().split("\\.");
Review comment:
Use `StringUtils.split(identifier.getName(), '.')` to avoid the regex match for better performance
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,561 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST =
+ new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST =
+ new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST =
+ new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable TableConfig config, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, config, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, @Nullable Schema schema,
+ DataSchema.ColumnDataType outputDataType) {
+ switch (expression.getType()) {
+ case LITERAL:
+ return new Pair<>(getLiteralSQL(expression.getLiteral(), true), false);
+ case IDENTIFIER: {
+ String[] parts = getIdentifierParts(expression.getIdentifier());
+ boolean hasJsonPathExpression = false;
+ String alias = expression.getIdentifier().getName();
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ // replace <column-name>.<json-path> with json_extract_scalar(<column-name>, '<json-path>', 'STRING', <JSON-null-value>)
+ Function jsonExtractScalarFunction = getJsonExtractFunction(parts, outputDataType);
+ expression.setIdentifier(null);
+ expression.setType(ExpressionType.FUNCTION);
+ expression.setFunctionCall(jsonExtractScalarFunction);
+ hasJsonPathExpression = true;
+ }
+ return new Pair<>(alias, hasJsonPathExpression);
+ }
+ case FUNCTION: {
+ Function function = expression.getFunctionCall();
+ List<Expression> operands = function.getOperands();
+
+ boolean hasJsonPathExpression = false;
+ StringBuffer alias = new StringBuffer();
+ if (function.getOperator().toUpperCase().equals("AS")) {
+ // We don't need to compute an alias for AS function since AS function defines its own alias.
+ hasJsonPathExpression = optimizeJsonIdentifier(operands.get(0), schema, outputDataType).getSecond();
+ alias.append(function.getOperands().get(1).getIdentifier().getName());
+ } else {
+ // For all functions besides AS function, process the operands and compute the alias.
+ alias.append(function.getOperator().toLowerCase(Locale.ROOT)).append("(");
+
+ // Output datatype of JSON_EXTRACT_SCALAR will depend upon the function within which json path expression appears.
+ outputDataType = getJsonExtractOutputDataType(function);
+
+ for (int i = 0; i < operands.size(); ++i) {
+ // recursively check to see if there is a <json-column>.<json-path> identifier in this expression.
+ Pair<String, Boolean> operandResult = optimizeJsonIdentifier(operands.get(i), schema, outputDataType);
+ hasJsonPathExpression |= operandResult.getSecond();
+ if (i > 0) {
+ alias.append(",");
+ }
+ alias.append(operandResult.getFirst());
+ }
+ alias.append(")");
+ }
+
+ return new Pair<>(alias.toString(), hasJsonPathExpression);
+ }
+ }
+
+ return new Pair<>("", false);
+ }
+
+ /**
+ * Example:
+ * Input:
+ * alias : "jsoncolumn.x.y.z",
+ * function: JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null')
+ * Output: AS(JSON_EXTRACT_SCALAR('jsoncolumn', 'x.y.z', 'STRING', 'null'), 'jsoncolumn.x.y.z')
+ *
+ * @return a Function with "AS" operator that wraps another function.
+ */
+ private static Function getAliasFunction(String alias, Function function) {
+ Function aliasFunction = new Function("AS");
+
+ List<Expression> operands = new ArrayList<>();
+ Expression expression = new Expression(ExpressionType.FUNCTION);
+ expression.setFunctionCall(function);
+ operands.add(expression);
+ operands.add(RequestUtils.createIdentifierExpression(alias));
+ aliasFunction.setOperands(operands);
+
+ return aliasFunction;
+ }
+
+ /**
+ * Example:
+ * Input : ["jsoncolumn", "x","y","z[2]"]
+ * Output: JSON_EXTRACT_SCALAR('jsoncolumn','$.x.y.z[2]','STRING','null')
+ *
+ * @param parts All the subparts of a fully qualified identifier (json path expression).
+ * @param dataType Output datatype of JSON_EXTRACT_SCALAR function.
+ * @return a Function with JSON_EXTRACT_SCALAR operator created using parts of fully qualified identifier name.
+ */
+ private static Function getJsonExtractFunction(String[] parts, DataSchema.ColumnDataType dataType) {
+ Function jsonExtractScalarFunction = new Function("JSON_EXTRACT_SCALAR");
+ List<Expression> operands = new ArrayList<>();
+ operands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(getJsonPath(parts, false))));
+ operands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(dataType.toString())));
+
+ operands.add(RequestUtils.createLiteralExpression(getDefaultNullValueForType(dataType)));
+ jsonExtractScalarFunction.setOperands(operands);
+ return jsonExtractScalarFunction;
+ }
+
+ /**
+ * Example 1:
+ * Input : "jsonColumn.name.first = 'daffy'"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.name.first\" = ''daffy''').
+ *
+ * Example 2:
+ * Input : "jsonColumn.id = 4"
+ * Output: "JSON_MATCH(jsonColumn, '\"$.id\" = 4')
+ */
+ private static void optimizeJsonPredicate(Expression expression, @Nullable TableConfig config,
+ @Nullable Schema schema) {
+ if (expression.getType() == ExpressionType.FUNCTION) {
+ Function function = expression.getFunctionCall();
+ String operator = function.getOperator();
+ FilterKind kind = FilterKind.valueOf(operator);
+ List<Expression> operands = function.getOperands();
+ switch (kind) {
+ case AND:
+ case OR: {
+ operands.forEach(operand -> optimizeJsonPredicate(operand, config, schema));
+ break;
+ }
+ case EQUALS:
+ case NOT_EQUALS:
+ case GREATER_THAN:
+ case GREATER_THAN_OR_EQUAL:
+ case LESS_THAN:
+ case LESS_THAN_OR_EQUAL: {
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ if (left.getType() == ExpressionType.IDENTIFIER && right.getType() == ExpressionType.LITERAL) {
+ String[] parts = getIdentifierParts(left.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(new StringLiteralAstNode(
+ getJsonPath(parts, true) + getOperatorSQL(kind) + getLiteralSQL(right.getLiteral(), false))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ left.clear();
+ left.setType(ExpressionType.FUNCTION);
+ left.setFunctionCall(getJsonExtractFunction(parts, getColumnTypeForLiteral(right.getLiteral())));
+ }
+ }
+ }
+ break;
+ }
+ case IS_NULL:
+ case IS_NOT_NULL: {
+ Expression operand = operands.get(0);
+ if (operand.getType() == ExpressionType.IDENTIFIER) {
+ String[] parts = getIdentifierParts(operand.getIdentifier());
+ if (parts.length > 1 && isValidJSONColumn(parts[0], schema)) {
+ if (isIndexedJSONColumn(parts[0], config)) {
+ Function jsonMatchFunction = new Function("JSON_MATCH");
+
+ List<Expression> jsonMatchFunctionOperands = new ArrayList<>();
+ jsonMatchFunctionOperands.add(RequestUtils.createIdentifierExpression(parts[0]));
+ jsonMatchFunctionOperands.add(RequestUtils.createLiteralExpression(
+ new StringLiteralAstNode(getJsonPath(parts, true) + getOperatorSQL(kind))));
+ jsonMatchFunction.setOperands(jsonMatchFunctionOperands);
+
+ expression.setFunctionCall(jsonMatchFunction);
+ } else {
+ operand.clear();
+ operand.setType(ExpressionType.FUNCTION);
+ operand.setFunctionCall(getJsonExtractFunction(parts, DataSchema.ColumnDataType.JSON));
+ }
+ }
+ }
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * @return A string array containing all the parts of an identifier. An identifier may have one or more parts that
+ * are joined together using <DOT>. For example the identifier "testTable.jsonColumn.name.first" consists up of
+ * "testTable" (name of table), "jsonColumn" (name of column), "name" (json path), and "first" (json path). The last
+ * two parts when joined together (name.first) represent a JSON path expression.
+ */
+ private static String[] getIdentifierParts(Identifier identifier) {
+ return identifier.getName().split("\\.");
+ }
+
+ /**
+ * Builds a json path expression when given identifier parts. For example,given [jsonColumn, name, first], this
+ * function will return "$.name.first" as json path expression.
+ * @param parts identifier parts
+ * @param applyDoubleQuote delimit json path with double quotes if true; otherwise, don't delimit json path.
+ * @return JSON path expression associated with the given identifier parts.
+ */
+ private static String getJsonPath(String[] parts, boolean applyDoubleQuote) {
+ StringBuilder builder = new StringBuilder();
+ if (applyDoubleQuote) {
+ builder.append("\"");
+ }
+
+ builder.append("$");
+ for (int i = 1; i < parts.length; i++) {
+ builder.append(".").append(parts[i]);
+ }
+
+ if (applyDoubleQuote) {
+ builder.append("\"");
+ }
+
+ return builder.toString();
+ }
+
+ /** @return true if specified column has column datatype of JSON; otherwise, return false */
+ private static boolean isValidJSONColumn(String columnName, @Nullable Schema schema) {
+ return schema != null && schema.hasColumn(columnName) && schema.getFieldSpecFor(columnName).getDataType()
+ .equals(FieldSpec.DataType.JSON);
+ }
+
+ /** @return true if specified column has a JSON Index. */
+ private static boolean isIndexedJSONColumn(String columnName, @Nullable TableConfig config) {
+ return config != null && config.getIndexingConfig().getJsonIndexColumns().contains(columnName);
Review comment:
`config.getIndexingConfig().getJsonIndexColumns()` might be `null` which will cause NPE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r656758865
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,516 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
Review comment:
I think this can be done by passing TableConfig into optimizer and then generating either JSON_MATCH or JSON_EXTRACT_SCALAR depending on whether index is present or not. We already generate both functions, so this should not be a big change.
` public void optimize(PinotQuery pinotQuery, TableConfig tableConfig, @Nullable Schema schema)`
Seems like segments on server not being in sync with table config is an independent issue that can be ignored here?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,516 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST = new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST = new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST = new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST = new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST = new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, true);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, false);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, false);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, false);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, Schema schema, DataSchema.ColumnDataType outputDataType,
+ boolean hasColumnAlias) {
Review comment:
Removed. This got left over from prior refactoring and is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (41c3f30) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/709abb01dabe8487856c1c2a2cffb01795f2a00c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (709abb0) will **decrease** coverage by `7.96%`.
> The diff coverage is `76.49%`.
> :exclamation: Current head 41c3f30 differs from pull request most recent head 2a0baab. Consider uploading reports for the commit 2a0baab to get more accurate results
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.48% 65.52% -7.97%
Complexity 91 91
============================================
Files 1490 1491 +1
Lines 73153 73399 +246
Branches 10525 10569 +44
============================================
- Hits 53756 48092 -5664
- Misses 15893 21902 +6009
+ Partials 3504 3405 -99
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.52% <76.49%> (+0.04%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `17.35% <0.00%> (-53.87%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.82% <73.91%> (-0.76%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `78.13% <78.13%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [352 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [709abb0...2a0baab](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8c317e4) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/70816838dda11f91708d482e8f4a400324e2f872?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7081683) will **increase** coverage by `31.84%`.
> The diff coverage is `81.60%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
=============================================
+ Coverage 41.81% 73.65% +31.84%
- Complexity 7 91 +84
=============================================
Files 1477 1478 +1
Lines 72800 73010 +210
Branches 10471 10511 +40
=============================================
+ Hits 30441 53779 +23338
+ Misses 39816 15747 -24069
- Partials 2543 3484 +941
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.94% <43.86%> (+0.13%)` | :arrow_up: |
| unittests | `65.62% <79.71%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.82% <73.91%> (+18.77%)` | :arrow_up: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `81.96% <81.96%> (ø)` | |
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `93.05% <100.00%> (+4.16%)` | :arrow_up: |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ot/segment/local/customobject/MinMaxRangePair.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9jdXN0b21vYmplY3QvTWluTWF4UmFuZ2VQYWlyLmphdmE=) | `75.86% <0.00%> (-24.14%)` | :arrow_down: |
| [.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `67.02% <0.00%> (-8.52%)` | :arrow_down: |
| [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `46.98% <0.00%> (-6.03%)` | :arrow_down: |
| [...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `63.85% <0.00%> (-6.03%)` | :arrow_down: |
| [...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `72.34% <0.00%> (-5.32%)` | :arrow_down: |
| [...core/query/executor/ServerQueryExecutorV1Impl.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=) | `81.42% <0.00%> (-4.38%)` | :arrow_down: |
| ... and [940 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [7081683...8c317e4](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8c317e4) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/70816838dda11f91708d482e8f4a400324e2f872?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7081683) will **increase** coverage by `0.13%`.
> The diff coverage is `43.86%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
+ Coverage 41.81% 41.94% +0.13%
Complexity 7 7
============================================
Files 1477 1478 +1
Lines 72800 73010 +210
Branches 10471 10511 +40
============================================
+ Hits 30441 30627 +186
+ Misses 39816 39813 -3
- Partials 2543 2570 +27
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.94% <43.86%> (+0.13%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `70.08% <0.00%> (-2.97%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `47.54% <47.54%> (ø)` | |
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `88.88% <100.00%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ot/segment/local/customobject/MinMaxRangePair.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9jdXN0b21vYmplY3QvTWluTWF4UmFuZ2VQYWlyLmphdmE=) | `75.86% <0.00%> (-24.14%)` | :arrow_down: |
| [.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `67.02% <0.00%> (-8.52%)` | :arrow_down: |
| [...nt/index/readers/sorted/SortedIndexReaderImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvc29ydGVkL1NvcnRlZEluZGV4UmVhZGVySW1wbC5qYXZh) | `91.83% <0.00%> (-6.13%)` | :arrow_down: |
| [...pache/pinot/core/query/optimizer/filter/Range.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvZmlsdGVyL1JhbmdlLmphdmE=) | `85.71% <0.00%> (-6.13%)` | :arrow_down: |
| [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `46.98% <0.00%> (-6.03%)` | :arrow_down: |
| [...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `63.85% <0.00%> (-6.03%)` | :arrow_down: |
| ... and [41 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [7081683...8c317e4](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: [WIP] Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (294b0e2) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/02d54b9d986a72e6155014b6e0245d10d85ff6c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (02d54b9) will **decrease** coverage by `7.76%`.
> The diff coverage is `72.45%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.17% 65.40% -7.77%
Complexity 12 12
============================================
Files 1451 1452 +1
Lines 71824 71990 +166
Branches 10412 10440 +28
============================================
- Hits 52554 47084 -5470
- Misses 15738 21495 +5757
+ Partials 3532 3411 -121
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.40% <72.45%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `82.43% <0.00%> (-8.11%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `72.04% <72.04%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../core/startree/plan/StarTreeTransformPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlVHJhbnNmb3JtUGxhbk5vZGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [341 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [02d54b9...294b0e2](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d7c4de8) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/929925568ce01b0c04d129253668a33e14ca1615?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9299255) will **decrease** coverage by `7.90%`.
> The diff coverage is `74.31%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.33% 65.43% -7.91%
Complexity 12 12
============================================
Files 1453 1454 +1
Lines 72013 72194 +181
Branches 10428 10464 +36
============================================
- Hits 52809 47238 -5571
- Misses 15667 21537 +5870
+ Partials 3537 3419 -118
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.43% <74.31%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `82.43% <0.00%> (-8.11%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.30% <69.56%> (-0.92%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `74.67% <74.67%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [345 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [9299255...d7c4de8](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648420886
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id > 10
+ * GROUP BY jsonColumn.id
+ * To: SELECT MIN(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity')) AS min(jsonColumn.id)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 10')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
Review comment:
SQL fragment ?
FWIW, fragment is a commonly used terminology for a unit of work in distributed query execution engines (see Presto docs for example).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648417499
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id > 10
+ * GROUP BY jsonColumn.id
+ * To: SELECT MIN(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity')) AS min(jsonColumn.id)
Review comment:
This doesn't look correct. When the user is not doing MIN() aggregation function anywhere, why are we rewriting the query to use MIN ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2cc4f2) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/709abb01dabe8487856c1c2a2cffb01795f2a00c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (709abb0) will **decrease** coverage by `31.72%`.
> The diff coverage is `14.34%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
=============================================
- Coverage 73.48% 41.75% -31.73%
+ Complexity 91 7 -84
=============================================
Files 1490 1491 +1
Lines 73153 73406 +253
Branches 10525 10572 +47
=============================================
- Hits 53756 30654 -23102
- Misses 15893 40177 +24284
+ Partials 3504 2575 -929
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.75% <14.34%> (+0.10%)` | :arrow_up: |
| unittests | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `70.08% <0.00%> (-22.50%)` | :arrow_down: |
| [...rc/main/java/org/apache/pinot/spi/data/Schema.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9TY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-75.32%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `11.81% <11.81%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `92.30% <71.42%> (-7.70%)` | :arrow_down: |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `66.26% <100.00%> (-4.96%)` | :arrow_down: |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/spi/config/table/TableType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1RhYmxlVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [944 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [709abb0...b2cc4f2](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-867230470
> Please also update the PR description queries where the column is missing in `json_match`
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia merged pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia merged pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r648423868
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id > 10
+ * GROUP BY jsonColumn.id
+ * To: SELECT MIN(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity')) AS min(jsonColumn.id)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 10')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
Review comment:
Can we add another example and corresponding test where json path expression appears in a transform function -- both scalar and nonscalar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: [WIP] Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (294b0e2) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/02d54b9d986a72e6155014b6e0245d10d85ff6c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (02d54b9) will **increase** coverage by `0.04%`.
> The diff coverage is `73.65%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
+ Coverage 73.17% 73.21% +0.04%
Complexity 12 12
============================================
Files 1451 1452 +1
Lines 71824 71990 +166
Branches 10412 10440 +28
============================================
+ Hits 52554 52709 +155
- Misses 15738 15748 +10
- Partials 3532 3533 +1
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.97% <45.50%> (+0.18%)` | :arrow_up: |
| unittests | `65.40% <72.45%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `72.67% <72.67%> (ø)` | |
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `90.54% <100.00%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `59.63% <0.00%> (-6.43%)` | :arrow_down: |
| [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `46.98% <0.00%> (-6.03%)` | :arrow_down: |
| [.../impl/dictionary/BaseOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvQmFzZU9mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `84.00% <0.00%> (-3.34%)` | :arrow_down: |
| [...gregation/function/StUnionAggregationFunction.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9TdFVuaW9uQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `71.42% <0.00%> (-2.86%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `87.35% <0.00%> (-2.30%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/data/TimeFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9UaW1lRmllbGRTcGVjLmphdmE=) | `88.63% <0.00%> (-2.28%)` | :arrow_down: |
| [...t/local/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `69.28% <0.00%> (-2.15%)` | :arrow_down: |
| ... and [23 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [02d54b9...294b0e2](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: [WIP] Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bea9c1c) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/79507da8edff75e931226f31fe349483f55e4406?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (79507da) will **decrease** coverage by `0.00%`.
> The diff coverage is `73.17%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
- Coverage 73.22% 73.21% -0.01%
Complexity 12 12
============================================
Files 1451 1452 +1
Lines 71824 71987 +163
Branches 10412 10439 +27
============================================
+ Hits 52591 52707 +116
- Misses 15700 15743 +43
- Partials 3533 3537 +4
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.87% <44.51%> (-0.04%)` | :arrow_down: |
| unittests | `65.46% <71.95%> (+0.05%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `72.15% <72.15%> (ø)` | |
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `90.54% <100.00%> (ø)` | |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [...readers/constant/ConstantMVForwardIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvY29uc3RhbnQvQ29uc3RhbnRNVkZvcndhcmRJbmRleFJlYWRlci5qYXZh) | `14.28% <0.00%> (-28.58%)` | :arrow_down: |
| [...nt/index/readers/ConstantValueBytesDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZUJ5dGVzRGljdGlvbmFyeS5qYXZh) | `23.80% <0.00%> (-4.77%)` | :arrow_down: |
| [...cal/startree/v2/builder/BaseSingleTreeBuilder.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9idWlsZGVyL0Jhc2VTaW5nbGVUcmVlQnVpbGRlci5qYXZh) | `88.39% <0.00%> (-4.47%)` | :arrow_down: |
| [...ot/segment/local/startree/OffHeapStarTreeNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS9PZmZIZWFwU3RhclRyZWVOb2RlLmphdmE=) | `81.48% <0.00%> (-3.71%)` | :arrow_down: |
| [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `80.64% <0.00%> (-3.23%)` | :arrow_down: |
| [.../predicate/NotEqualsPredicateEvaluatorFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL05vdEVxdWFsc1ByZWRpY2F0ZUV2YWx1YXRvckZhY3RvcnkuamF2YQ==) | `75.32% <0.00%> (-2.60%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [79507da...bea9c1c](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (645c582) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/709abb01dabe8487856c1c2a2cffb01795f2a00c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (709abb0) will **increase** coverage by `0.08%`.
> The diff coverage is `81.22%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
+ Coverage 73.48% 73.56% +0.08%
Complexity 91 91
============================================
Files 1490 1491 +1
Lines 73153 73393 +240
Branches 10525 10567 +42
============================================
+ Hits 53756 53994 +238
+ Misses 15893 15880 -13
- Partials 3504 3519 +15
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.66% <42.44%> (+0.01%)` | :arrow_up: |
| unittests | `65.54% <76.73%> (+0.07%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.82% <73.91%> (-0.76%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `80.67% <80.67%> (ø)` | |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `71.35% <100.00%> (+0.12%)` | :arrow_up: |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...readers/constant/ConstantMVForwardIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvY29uc3RhbnQvQ29uc3RhbnRNVkZvcndhcmRJbmRleFJlYWRlci5qYXZh) | `14.28% <0.00%> (-28.58%)` | :arrow_down: |
| [...me/impl/dictionary/IntOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvSW50T25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `78.31% <0.00%> (-4.82%)` | :arrow_down: |
| [...nt/index/readers/ConstantValueBytesDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQ29uc3RhbnRWYWx1ZUJ5dGVzRGljdGlvbmFyeS5qYXZh) | `23.80% <0.00%> (-4.77%)` | :arrow_down: |
| [...cal/startree/v2/builder/BaseSingleTreeBuilder.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9idWlsZGVyL0Jhc2VTaW5nbGVUcmVlQnVpbGRlci5qYXZh) | `89.23% <0.00%> (-4.49%)` | :arrow_down: |
| [...ot/segment/local/startree/OffHeapStarTreeNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS9PZmZIZWFwU3RhclRyZWVOb2RlLmphdmE=) | `81.48% <0.00%> (-3.71%)` | :arrow_down: |
| [...core/startree/operator/StarTreeFilterOperator.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9vcGVyYXRvci9TdGFyVHJlZUZpbHRlck9wZXJhdG9yLmphdmE=) | `92.56% <0.00%> (-2.48%)` | :arrow_down: |
| ... and [20 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [709abb0...645c582](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r650319742
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
Review comment:
Fixed.
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
Review comment:
@mcvsubbu Good idea. Done.
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
Review comment:
Working on this.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
Review comment:
The current convention is that `SELECT max(intColumn - 5) FROM testTable` will have alias `max(minus(intColumn,'5'))`, so this is just extending the current convention to use json path expressions.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id > 10
+ * GROUP BY jsonColumn.id
+ * To: SELECT MIN(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity')) AS min(jsonColumn.id)
Review comment:
Fixed.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id > 10
+ * GROUP BY jsonColumn.id
+ * To: SELECT MIN(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity')) AS min(jsonColumn.id)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 10')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
Review comment:
I am not surprised that this term is overloaded, but common usage of term "sql fragment" still appears to refer to a part of SQL statement from what I can see from a google search.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,436 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity'),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id > 10
+ * GROUP BY jsonColumn.id
+ * To: SELECT MIN(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', '-Infinity')) AS min(jsonColumn.id)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 10')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
Review comment:
Done.
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
Review comment:
There is already a test case for this.
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
+ @Test
+ public void testJsonSelect() {
+ // SELECT using a simple json path expression.
+ BrokerRequest sqlBrokerRequest1 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.x FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.x>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.x))]))");
+
+ // SELECT using json path expressions with array addressing.
+ BrokerRequest sqlBrokerRequest2 = SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.data[0][1].a.b[0] FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.data[0][1].a.b[0]))]))");
+
+ // SELECT using json path expressions within double quotes.
+ BrokerRequest sqlBrokerRequest3 = SQL_COMPILER.compileToBrokerRequest("SELECT \"jsonColumn.a.b.c[0][1][2][3].d.e.f[0].g\" FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.data[0][1].a.b[0]>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))]))");
+ }
+
+ /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH function. */
+ @Test
+ public void testJsonFilter() {
+ // String literal
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.name.first = 'daffy'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.name.first\" = 'daffy'>)]))");
+
+ // Numerical literal
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT * FROM testTable WHERE jsonColumn.id = 101");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+ }
+
+ /** Test that a json path expression in GROUP BY clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupBy() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.id, count(*) FROM testTable GROUP BY jsonColumn.id");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
+ Assert.assertEquals(pinotQuery.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.id))]))");
+
+ Assert.assertEquals(pinotQuery.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+ }
+
+ /** Test that a json path expression in HAVING clause is properly converted into a JSON_EXTRACT_SCALAR function. */
+ @Test
+ public void testJsonGroupByHaving() {
+
+ // Test HAVING clause with a STRING value extracted using json path expression.
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.name.last, count(*) FROM testTable GROUP BY jsonColumn.name.last HAVING jsonColumn.name.last = 'mouse'");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.name.last))]))");
+
+ Assert.assertEquals(pinotQuery1.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+
+ Assert.assertEquals(pinotQuery1.getHavingExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:LITERAL, literal:<Literal stringValue:mouse>)]))");
+
+ // Test HAVING clause with a DOUBLE value extract using json path expression.
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT jsonColumn.name.last, count(*) FROM testTable GROUP BY jsonColumn.name.last HAVING jsonColumn.name.last = 'mouse'");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.name.last))]))");
+
+ Assert.assertEquals(pinotQuery2.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+
+ Assert.assertEquals(pinotQuery2.getHavingExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:LITERAL, literal:<Literal stringValue:mouse>)]))");
+ }
+
+ /** Test a complex SQL statement with json path expression in SELECT, WHERE, and GROUP BY clauses. */
+ @Test
+ public void testJsonSelectFilterGroupBy() {
+ BrokerRequest sqlBrokerRequest = SQL_COMPILER.compileToBrokerRequest(
+ "SELECT jsonColumn.name.last, count(*) FROM testTable WHERE jsonColumn.id = 101 GROUP BY jsonColumn.name.last");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
+ Assert.assertEquals(pinotQuery.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)])), Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn.name.last))]))");
+
+ Assert.assertEquals(pinotQuery.getFilterExpression().toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_MATCH, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:\"$.id\" = 101>)]))");
+
+ Assert.assertEquals(pinotQuery.getGroupByList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.last>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))");
+ }
+
+ /** Test an aggregation function over json path expression in SELECT clause. */
+ @Test
+ public void testStringFunctionOverJsonPathSelectExpression() {
+ BrokerRequest sqlBrokerRequest =
+ SQL_COMPILER.compileToBrokerRequest("SELECT UPPER(jsonColumn.name.first) FROM testTable");
+ PinotQuery pinotQuery = sqlBrokerRequest.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery, SCHEMA);
+
+ Assert.assertEquals(pinotQuery.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:UPPER, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.name.first>), Expression(type:LITERAL, literal:<Literal stringValue:STRING>), Expression(type:LITERAL, literal:<Literal stringValue:null>)]))])), Expression(type:IDENTIFIER, identifier:Identifier(name:upper(jsonColumn.name.first)))]))");
+ }
+
+ /** Test a numerical function over json path expression in SELECT clause. */
+ @Test
+ public void testNumericalFunctionOverJsonPathSelectExpression() {
+
+ // Test without user-specified alias.
+ BrokerRequest sqlBrokerRequest1 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT MAX(jsonColumn.id) FROM testTable");
+ PinotQuery pinotQuery1 = sqlBrokerRequest1.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery1, SCHEMA);
+
+ Assert.assertEquals(pinotQuery1.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:MAX, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:DOUBLE>), Expression(type:LITERAL, literal:<Literal doubleValue:-Infinity>)]))])), Expression(type:IDENTIFIER, identifier:Identifier(name:max(jsonColumn.id)))]))");
+
+ // Test with user-specified alias.
+ BrokerRequest sqlBrokerRequest2 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT MAX(jsonColumn.id) AS x FROM testTable");
+ PinotQuery pinotQuery2 = sqlBrokerRequest2.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery2, SCHEMA);
+
+ Assert.assertEquals(pinotQuery2.getSelectList().get(0).toString(),
+ "Expression(type:FUNCTION, functionCall:Function(operator:AS, operands:[Expression(type:FUNCTION, functionCall:Function(operator:MAX, operands:[Expression(type:FUNCTION, functionCall:Function(operator:JSON_EXTRACT_SCALAR, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:jsonColumn)), Expression(type:LITERAL, literal:<Literal stringValue:$.id>), Expression(type:LITERAL, literal:<Literal stringValue:DOUBLE>), Expression(type:LITERAL, literal:<Literal doubleValue:-Infinity>)]))])), Expression(type:IDENTIFIER, identifier:Identifier(name:x))]))");
+
+ // Test with nested function calls (minus function being used within max function).
+ BrokerRequest sqlBrokerRequest3 =
+ SQL_COMPILER.compileToBrokerRequest("SELECT MAX(jsonColumn.id - 5) FROM testTable");
+ PinotQuery pinotQuery3 = sqlBrokerRequest3.getPinotQuery();
+ OPTIMIZER.optimize(pinotQuery3, SCHEMA);
+
Review comment:
Added.
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizerTest.java
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.query.optimizer.QueryOptimizer;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class JsonStatementOptimizerTest {
+ private static final QueryOptimizer OPTIMIZER = new QueryOptimizer();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+ private static final Schema SCHEMA =
+ new Schema.SchemaBuilder().setSchemaName("testTable").addSingleValueDimension("intColumn", FieldSpec.DataType.INT)
+ .addSingleValueDimension("longColumn", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("stringColumn", FieldSpec.DataType.STRING)
+ .addMultiValueDimension("jsonColumn", FieldSpec.DataType.JSON).build();
+
+ /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within an AS function. */
Review comment:
Added query execution test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r656506126
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,516 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
+ *
+ * Example 1:
+ * From : SELECT jsonColumn.name.first
+ * FROM testTable
+ * WHERE jsonColumn.name.first IS NOT NULL
+ * TO : SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.first', 'STRING', 'null') AS jsonColum.name.first
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" IS NOT NULL')
+ *
+ * Output datatype of any json path expression is 'STRING'. However, if json path expression appears as an argument to
+ * a numerical function, then output of json path expression is set to 'DOUBLE' as shown in the example below.
+ *
+ * Example 2:
+ * From: SELECT MIN(jsonColumn.id - 5)
+ * FROM testTable
+ * WHERE jsonColumn.id IS NOT NULL
+ * To: SELECT MIN(MINUS(JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'DOUBLE', Double.NEGATIVE_INFINITY),5)) AS min(minus(jsonColum.id, '5'))
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.id" IS NOT NULL')
+ *
+ * Example 3:
+ * From: SELECT jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE jsonColumn.name.first = 'Daffy' OR jsonColumn.id = 101
+ * GROUP BY jsonColumn.id
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null') AS jsonColumn.id, count(*)
+ * FROM testTable
+ * WHERE JSON_MATCH('"$.name.first" = ''Daffy''') OR JSON_MATCH('"$.id" = 101')
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.id', 'STRING', 'null');
+ *
+ * Example 4:
+ * From: SELECT jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY jsonColumn.name.last
+ * HAVING jsonColumn.name.last = 'mouse'
+ * To: SELECT JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') AS jsonColumn.name.last, count(*)
+ * FROM testTable
+ * GROUP BY JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null')
+ * HAVING JSON_EXTRACT_SCALAR(jsonColumn, '$.name.last', 'STRING', 'null') = 'mouse'
+ *
+ * Notes:
+ * 1) In a filter expression, if json path appears on the left-hand side, the right-hand side must be a literal. In
+ * future this can be changed to have any expression on the right-hand side by implementing a function that would
+ * convert any {@link Expression} into SQL fragment that can be used in JSON_MATCH. Currently only literals are
+ * converted into SQL fragments {see @link #getLiteralSQL} function.
+ * 2) In WHERE clause each json path expression will be replaced with a JSON_MATCH function. If there are multiple
+ * json path expressions, they will be replaced by multiple JSON_MATCH functions. We currently don't fold multiple
+ * JSON_MATCH functions into a single JSON_MATCH_FUNCTION.
+ */
+public class JsonStatementOptimizer implements StatementOptimizer {
+
+ /**
+ * Maintain a list of numerical functions that requiring json path expression to output numerical values. This allows
+ * us to implicitly convert the output of json path expression to DOUBLE. TODO: There are better ways of doing this
+ * if we were to move to a new storage (currently STRING) for JSON or functions were to pre-declare their input
+ * data types.
+ */
+ private static Set<String> numericalFunctions = getNumericalFunctionList();
+
+ /**
+ * A list of functions that require json path expression to output LONG value. This allows us to implicitly convert
+ * the output of json path expression to LONG.
+ */
+ private static Set<String> datetimeFunctions = getDateTimeFunctionList();
+
+ /**
+ * Null value constants for different column types. Used while rewriting json path expression to JSON_EXTRACT_SCALAR function.
+ */
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_INT_AST = new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_INT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_LONG_AST = new IntegerLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT_AST = new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_FLOAT);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE_AST = new FloatingPointLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_DOUBLE);
+ private static LiteralAstNode DEFAULT_DIMENSION_NULL_VALUE_OF_STRING_AST = new StringLiteralAstNode(FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
+
+ @Override
+ public void optimize(PinotQuery query, @Nullable Schema schema) {
+ // In SELECT clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function with an alias.
+ List<Expression> expressions = query.getSelectList();
+ for (Expression expression : expressions) {
+ Pair<String, Boolean> result = optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, true);
+ if (expression.getType() == ExpressionType.FUNCTION && !expression.getFunctionCall().getOperator().equals("AS")
+ && result.getSecond()) {
+ // Since this is not an AS function (user-specified alias) and the function or its arguments contain json path
+ // expression, set an alias for the expression after replacing json path expression with JSON_EXTRACT_SCALAR
+ // function.
+ Function aliasFunction = getAliasFunction(result.getFirst(), expression.getFunctionCall());
+ expression.setFunctionCall(aliasFunction);
+ }
+ }
+
+ // In WHERE clause, replace JSON path expressions with JSON_MATCH function.
+ Expression filter = query.getFilterExpression();
+ if (filter != null) {
+ optimizeJsonPredicate(filter, schema);
+ }
+
+ // In GROUP BY clause, replace JSON path expressions with JSON_EXTRACT_SCALAR function without an alias.
+ expressions = query.getGroupByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, false);
+ }
+ }
+
+ // In ORDER BY clause, replace JSON path expression with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ expressions = query.getOrderByList();
+ if (expressions != null) {
+ for (Expression expression : expressions) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, false);
+ }
+ }
+
+ // In HAVING clause, replace JSON path expressions with JSON_EXTRACT_SCALAR. This expression must match the
+ // corresponding SELECT list expression except for the alias.
+ Expression expression = query.getHavingExpression();
+ if (expression != null) {
+ optimizeJsonIdentifier(expression, schema, DataSchema.ColumnDataType.STRING, false);
+ }
+ }
+
+ /**
+ * Replace an json path expression with an aliased JSON_EXTRACT_SCALAR function.
+ * @param expression input expression to rewrite into JSON_EXTRACT_SCALAR function if the expression is json path.
+ * @param outputDataType to keep track of output datatype of JSON_EXTRACT_SCALAR function which depends upon the outer
+ * function that json path expression appears in.
+ * @return A {@link Pair} of values where the first value is alias for the input expression and second
+ * value indicates whether json path expression was found (true) or not (false) in the expression.
+ */
+ private static Pair<String, Boolean> optimizeJsonIdentifier(Expression expression, Schema schema, DataSchema.ColumnDataType outputDataType,
+ boolean hasColumnAlias) {
Review comment:
`hasColumnAlias` is unused?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/JsonStatementOptimizer.java
##########
@@ -0,0 +1,516 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.optimizer.statement;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.function.scalar.ArithmeticFunctions;
+import org.apache.pinot.common.function.scalar.DateTimeFunctions;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FloatingPointLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IntegerLiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.StringLiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.Pair;
+
+
+/**
+ * This class will rewrite a query that has json path expressions into a query that uses JSON_EXTRACT_SCALAR and
+ * JSON_MATCH functions.
Review comment:
We can only rewrite the query to `JSON_MATCH` when json index is available. One option is to read that info from the table config, but segments on server might not be always in-sync with the table config
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -163,7 +163,7 @@ protected void processSegments(int taskIndex) {
LOGGER.error("Caught exception while executing operator of index: {} (query: {})", operatorIndex, _queryContext,
e);
_blockingQueue.offer(new IntermediateResultsBlock(e));
- return;
+ throw e;
Review comment:
(Critical) Why changing this? The exception is already embedded in the results block, and we don't want to kill the working thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657509274
##########
File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
##########
@@ -345,23 +345,29 @@ private BrokerResponseNative handleSQLRequest(long requestId, String query, Json
if (offlineTableName != null && realtimeTableName != null) {
// Hybrid
offlineBrokerRequest = getOfflineBrokerRequest(brokerRequest);
- _queryOptimizer.optimize(offlineBrokerRequest.getPinotQuery(), schema);
+ _queryOptimizer.optimize(offlineBrokerRequest.getPinotQuery(),
+ _tableCache.getTableConfig(TableNameBuilder.OFFLINE.tableNameWithType(rawTableName)), schema);
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#issuecomment-852478013
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6998](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (41c3f30) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/709abb01dabe8487856c1c2a2cffb01795f2a00c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (709abb0) will **increase** coverage by `0.07%`.
> The diff coverage is `80.07%`.
> :exclamation: Current head 41c3f30 differs from pull request most recent head 2a0baab. Consider uploading reports for the commit 2a0baab to get more accurate results
[](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6998 +/- ##
============================================
+ Coverage 73.48% 73.56% +0.07%
Complexity 91 91
============================================
Files 1490 1491 +1
Lines 73153 73399 +246
Branches 10525 10569 +44
============================================
+ Hits 53756 53994 +238
+ Misses 15893 15885 -8
- Partials 3504 3520 +16
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.78% <40.63%> (+0.13%)` | :arrow_up: |
| unittests | `65.52% <76.49%> (+0.04%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `91.82% <73.91%> (-0.76%)` | :arrow_down: |
| [...ry/optimizer/statement/JsonStatementOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvc3RhdGVtZW50L0pzb25TdGF0ZW1lbnRPcHRpbWl6ZXIuamF2YQ==) | `79.53% <79.53%> (ø)` | |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `71.50% <100.00%> (+0.28%)` | :arrow_up: |
| [...che/pinot/core/query/optimizer/QueryOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9vcHRpbWl6ZXIvUXVlcnlPcHRpbWl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...nction/DistinctCountBitmapAggregationFunction.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9EaXN0aW5jdENvdW50Qml0bWFwQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `46.39% <0.00%> (-6.19%)` | :arrow_down: |
| [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `46.98% <0.00%> (-6.03%)` | :arrow_down: |
| [...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `63.85% <0.00%> (-6.03%)` | :arrow_down: |
| [...me/impl/dictionary/IntOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvSW50T25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `78.31% <0.00%> (-4.82%)` | :arrow_down: |
| [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `80.00% <0.00%> (-2.06%)` | :arrow_down: |
| [...e/pinot/segment/local/io/util/PinotDataBitSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby91dGlsL1Bpbm90RGF0YUJpdFNldC5qYXZh) | `96.03% <0.00%> (-1.59%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/apache/incubator-pinot/pull/6998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [709abb0...2a0baab](https://codecov.io/gh/apache/incubator-pinot/pull/6998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] amrishlal commented on a change in pull request #6998: Support json path expressions in query.
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #6998:
URL: https://github.com/apache/incubator-pinot/pull/6998#discussion_r657509351
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/QueryOptimizer.java
##########
@@ -61,16 +67,24 @@ public void optimize(BrokerRequest brokerRequest, @Nullable Schema schema) {
}
}
- /**
- * Optimizes the given SQL query.
- */
+ /** Optimizes the given SQL query. */
public void optimize(PinotQuery pinotQuery, @Nullable Schema schema) {
+ optimize(pinotQuery, null, schema);
+ }
+
+ /** Optimizes the given SQL query. */
+ public void optimize(PinotQuery pinotQuery, @Nullable TableConfig config, @Nullable Schema schema) {
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org