You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/23 22:23:02 UTC

[GitHub] [pinot] richardstartin opened a new pull request #7820: push JSON Path evaluation down to storage layer

richardstartin opened a new pull request #7820:
URL: https://github.com/apache/pinot/pull/7820


   Pushes JSON path evaluation down to the storage layer (giving direct access to dictionaries and forward index) which avoids various intermediate materialisations of strings, byte arrays and so on. Creates SPI to make the evaluation logic pluggable. Same mechanism could be extensible to regular expressions in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759974790



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       I'm not sure I agree about that, I don't think that this test was stable w.r.t. extension of the test setup. 
   
   In order to add columns to the test setup (e.g. to test a RAW JSON column, which was not previously tested, and is now) `expecteds1` would always need to be extended to include every column added. So there is a choice: add the values to `expected1`, duplicate the whole test _setup_ to add the new columns for a new test, or control what is projected to keep this test stable. Since this is a test for selection and not projection, I opted to do the latter. 
   
   SO do you want me to duplicate the test setup so I can add tests for querying a RAW JSON column, or change the query back to `select *` and add all new columns to `expected1`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985445101


   @Jackie-Jiang please see https://github.com/apache/pinot/pull/7820/commits/83dc232fc5d89dc1331437bc444512bfcd519eec - it removes `pushDown` from the `Block` interface. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760596693



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       @amrishlal I split the tests so this test only tests JSON path expressions and not the extract scalar function, but moved the setup into a base class so it can be reused. Does this resolve this issue?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5e1cfe) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `57.12%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head a5e1cfe differs from pull request most recent head db04a8c. Consider uploading reports for the commit db04a8c to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.53%   -57.13%     
   + Complexity     4088       80     -4008     
   =============================================
     Files          1579     1537       -42     
     Lines         80851    79512     -1339     
     Branches      12017    11924       -93     
   =============================================
   - Hits          57938    11555    -46383     
   - Misses        19017    67103    +48086     
   + Partials       3896      854     -3042     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.53% <0.00%> (-0.11%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/core/common/Block.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vQmxvY2suamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `0.00% <0.00%> (-69.24%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1259 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...db04a8c](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4d6aeff) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `40.86%`.
   > The diff coverage is `20.76%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   30.87%   -40.87%     
   =============================================
     Files          1579     1572        -7     
     Lines         80788    80987      +199     
     Branches      12002    12081       +79     
   =============================================
   - Hits          57953    25003    -32950     
   - Misses        18954    53840    +34886     
   + Partials       3881     2144     -1737     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `29.11% <16.38%> (-0.39%)` | :arrow_down: |
   | integration2 | `27.78% <11.23%> (-0.18%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `16.96% <16.96%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `12.50% <19.44%> (-37.03%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `61.07% <39.02%> (-28.89%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `82.23% <40.00%> (-13.98%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `46.66% <40.00%> (-13.34%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `53.48% <40.00%> (-24.78%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1064 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...4d6aeff](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (73588e7) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `57.10%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head 73588e7 differs from pull request most recent head 909cd9c. Consider uploading reports for the commit 909cd9c to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.55%   -57.11%     
   + Complexity     4088       80     -4008     
   =============================================
     Files          1579     1537       -42     
     Lines         80851    79480     -1371     
     Branches      12017    11924       -93     
   =============================================
   - Hits          57938    11571    -46367     
   - Misses        19017    67049    +48032     
   + Partials       3896      860     -3036     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.55% <0.00%> (-0.08%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/core/common/Block.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vQmxvY2suamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `0.00% <0.00%> (-69.24%)` | :arrow_down: |
   | [...edicate/BaseDictionaryBasedPredicateEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0Jhc2VEaWN0aW9uYXJ5QmFzZWRQcmVkaWNhdGVFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (-48.00%)` | :arrow_down: |
   | [...rator/filter/predicate/BasePredicateEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0Jhc2VQcmVkaWNhdGVFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [...predicate/BaseRawValueBasedPredicateEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0Jhc2VSYXdWYWx1ZUJhc2VkUHJlZGljYXRlRXZhbHVhdG9yLmphdmE=) | `0.00% <0.00%> (-86.37%)` | :arrow_down: |
   | [...ter/predicate/EqualsPredicateEvaluatorFactory.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvcHJlZGljYXRlL0VxdWFsc1ByZWRpY2F0ZUV2YWx1YXRvckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (-85.25%)` | :arrow_down: |
   | ... and [1270 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...909cd9c](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982140750

> Conceptually, we are moving from a Column to Column + path but modelling the path as an evaluator. But the query layer is deciding the evaluator instead of the storage layer.
>
> The evaluator seems to be set at the JVM level but in practice, it's possible for each table to have a different evaluator and in fact, it's possible that each segment can have a different evaluator.

The query layer does not decide, the query layer just _produces_ the evaluator from the query. It does not know what evaluator implementation it is producing, just whatever has been registered. It just won’t push the evaluator down unless the identifier is a column identifier rather than an expression.

The evaluator, having captured information from the query, is pushed down to the storage layer. The storage layer then presents storage layer capabilities (I.e. dictionary + forward index) to the evaluator, which can interrogate and choose what to do with them.

Think of the evaluator as a _mediator_ between the query and storage layers for which any implementation can be registered via SPI. It encapsulates the logic of the function, and needs to know all relevant storage capabilities present in the deployment. The right evaluator to register for a given Pinot _deployment_ is the one that matches all storage capabilities in the deployment.

The evaluator must decide how to evaluate its function based on what storage capabilities it is presented with in evaluateBlock. If evaluateBlock is presented with, say, a BytesDictionary, it must be able handle extracting JSON encoded as bytes from the dictionary and evaluating a JSON path against it, just as it must be able to handle reading JSON from a raw forward index, or from whatever ForwardIndexReader from whatever storage plugin it is presented with.

To illustrate the evaluator’s responsibility to understand the storage, the DefaultJsonPathEvaluator in this PR knows how to work with all storage capabilities capable of storing JSON currently present in Pinot (dictionarised string, bytes, raw string, bytes, JSON type column or not) but adding a storage plugin means creating a new evaluator which understands the custom storage but delegating to a default evaluator for everything else, and registering it via the SPI.

This has the ergonomic benefit of being able to implement storage plugins which deviate from the ForwardIndexReader interface. Completely new access patterns and the necessary interfaces can be implemented without changing the ForwardIndexReader interface:
1. Implement new ForwardIndexReader with a previously inconceivable API
2. Implement an evaluator which uses the new API for instances of that ForwardIndexReader, otherwise delegate to default implementation
3. Register the evaluator
4. (Optional) later enhance the ForwardIndexReader interface with method supporting a tried and proven access pattern, explain to everybody why it’s a good idea with data to refer to

This circumnavigates the need to form design consensus before having a proven storage implementation and should reduce the number of changes to the ForwardIndexReader interface to those which have already been **proven to make sense**, while never touching the query layer once the relevant TransformFunction has been modified to push an evaluator down.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982140750

To illustrate the evaluator’s responsibility to understand the storage, the DefaultJsonPathEvaluator in this PR knows how to work with all storage capabilities for storing JSON currently present in Pinot (dictionarised string, bytes, raw string, bytes, JSON type column or not) but adding a storage plugin requires creating a new evaluator which understands the custom storage but delegates to a default evaluator for everything else, and this evaluator must be registered via the SPI.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e362ace) into [master](https://codecov.io/gh/apache/pinot/commit/eff535134290a61b52a68a813a6022519d7ad1e0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff5351) will **decrease** coverage by `57.10%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.55%   -57.11%     
   + Complexity     4086       80     -4006     
   =============================================
     Files          1579     1536       -43     
     Lines         80839    79465     -1374     
     Branches      12010    11916       -94     
   =============================================
   - Hits          57930    11565    -46365     
   - Misses        19012    67050    +48038     
   + Partials       3897      850     -3047     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.55% <0.00%> (-0.04%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.97%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1257 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff5351...e362ace](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] kishoreg commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

kishoreg commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-983856821


   Thanks @richardstartin. I did pull the branch and reviewed the code. Given the state of the current code, I think what you have done is probably the least invasive change. I will list my wishful state to get to here
   ```
   public interface ForwardIndexReader<T extends ForwardIndexReaderContext> extends Closeable {
   
     boolean isNested();
   
     boolean isStructured();
   
     List<String> getNestedFields();
   
     ForwardIndexReader<T> getReader(String nestedField); or 
     ForwardIndexReader<T> getReader(Extractor/Evaluator e); or
     ForwardIndexReader<T> getReader(Function pushDownFunction); or
     ForwardIndexReader<T> getReader(Selector selector); or
   }
   ```
   
   But passing parameters down the Block abstraction is non-trivial given the state of the current code. 
   
   One request from my side would be to add javadoc the newly added methods with the evaluator as Beta that we will mark it stable once we get everything working end to end.
   
   Thanks again for taking the time to explain 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759518927



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/TransformBlock.java
##########
@@ -78,4 +79,54 @@ public BlockDocIdValueSet getBlockDocIdValueSet() {
   public BlockMetadata getMetadata() {
     throw new UnsupportedOperationException();
   }
+
+  @Override
+  public void pushDown(String column, TransformEvaluator evaluator, int[] buffer) {
+    _projectionBlock.pushDown(column, evaluator, buffer);

Review comment:
       If there is a column name it must be an identifier. I had previously only implemented this for `ProjectionBlock` for this reason though.

##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/TransformEvaluator.java
##########
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator;
+
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+
+
+public interface TransformEvaluator {
+  /**
+   * Evaluate the JSON path and fill the value buffer

Review comment:
       As above.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759178462



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       I just pushed something which should address your concerns. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982140750

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985152221


   > Could this be handled differently without the cache by reordering the execution graph to deduplicate common subexpressions?
   
   I agree. Ideally we should deduplicate the expression during the query planning, and have only one transform function for each unique expression. Then we can remove the cache and avoid all the unnecessary overhead.
   
   Towards this direction, I agree we should maintain the buffer within the transform layer and push the buffer down through the data fetcher. We should think of a smooth way to change the interface and move to the end state where all the transform functions (with or without an evaluator) can pass down the buffer to fill the values.
   
   For now, we might just add some read methods into the `ProjectionBlock` so that it can be used by the `TransformFunction`? Adding these methods to the `Block` interface might not match our end goal.
   
   Do you see a path how we can move to the end state without making temporary changes to the interfaces?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bcb55a2) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `6.80%`.
   > The diff coverage is `32.45%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.73%   64.93%   -6.81%     
   - Complexity     4087     4093       +6     
   ============================================
     Files          1579     1536      -43     
     Lines         80788    79547    -1241     
     Branches      12002    11907      -95     
   ============================================
   - Hits          57953    51651    -6302     
   - Misses        18954    24223    +5269     
   + Partials       3881     3673     -208     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.27% <32.45%> (-0.41%)` | :arrow_down: |
   | unittests2 | `14.50% <0.00%> (-0.09%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `23.56% <23.56%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `17.56% <28.70%> (-31.96%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `57.57% <57.57%> (ø)` | |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `88.81% <60.00%> (-7.40%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.71% <60.97%> (-4.25%)` | :arrow_down: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [378 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...bcb55a2](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad5aa15) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `57.22%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   14.50%   -57.23%     
   + Complexity     4087       80     -4007     
   =============================================
     Files          1579     1536       -43     
     Lines         80788    79466     -1322     
     Branches      12002    11893      -109     
   =============================================
   - Hits          57953    11529    -46424     
   - Misses        18954    67086    +48132     
   + Partials       3881      851     -3030     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.50% <0.00%> (-0.08%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1260 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...ad5aa15](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a4b6d90) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `42.38%`.
   > The diff coverage is `16.38%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   29.34%   -42.39%     
   =============================================
     Files          1579     1572        -7     
     Lines         80788    80987      +199     
     Branches      12002    12081       +79     
   =============================================
   - Hits          57953    23769    -34184     
   - Misses        18954    55113    +36159     
   + Partials       3881     2105     -1776     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `29.34% <16.38%> (-0.15%)` | :arrow_down: |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `14.13% <14.13%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `10.81% <14.81%> (-38.72%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `58.57% <29.26%> (-31.39%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `78.94% <30.00%> (-17.27%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `40.00% <30.00%> (-20.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `48.83% <30.00%> (-29.43%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1109 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...a4b6d90](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982151840

> Conceptually, we are moving from a Column to Column + path but modelling the path as an evaluator. But the query layer is deciding the evaluator instead of the storage layer.

Specifically on this, not really, the model has the potential to be far more generic than that. It’s more going from column to column + parameters + arbitrary sequence of storage layer API calls.

This is good, because within the context of an evaluator, the most efficient API calls can be made. Case in point: in DefaultJsonPathEvaluator it’s possible to avoid converting byte[] to String. It would also be possible to use hypothetical new zero copy (but unsafe) methods to read bytes on the basis that we don’t need to create a vector of JSON records, but a vector of JSON evaluation results. Individual evaluator implementations can be arbitrarily efficient (and complex) in a way that prevents the query layer from ever seeing any of this complexity, keeping the query layer clean just as it avoids trying to model all of the complex access patterns we might like in the storage layer with a single interface.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7a24e82) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `57.18%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   14.55%   -57.19%     
   + Complexity     4087       80     -4007     
   =============================================
     Files          1579     1536       -43     
     Lines         80788    79463     -1325     
     Branches      12002    11893      -109     
   =============================================
   - Hits          57953    11564    -46389     
   - Misses        18954    67042    +48088     
   + Partials       3881      857     -3024     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.55% <0.00%> (-0.03%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1260 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...7a24e82](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad5aa15) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `6.78%`.
   > The diff coverage is `33.02%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.73%   64.95%   -6.79%     
   - Complexity     4087     4089       +2     
   ============================================
     Files          1579     1536      -43     
     Lines         80788    79466    -1322     
     Branches      12002    11893     -109     
   ============================================
   - Hits          57953    51616    -6337     
   - Misses        18954    24171    +5217     
   + Partials       3881     3679     -202     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.29% <33.02%> (-0.39%)` | :arrow_down: |
   | unittests2 | `14.50% <0.00%> (-0.08%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `23.20% <23.20%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `17.56% <28.70%> (-31.96%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `57.57% <57.57%> (ø)` | |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.35% <58.53%> (-4.61%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `88.81% <60.00%> (-7.40%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [372 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...ad5aa15](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a4b6d90) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `40.79%`.
   > The diff coverage is `20.76%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   30.93%   -40.80%     
   =============================================
     Files          1579     1572        -7     
     Lines         80788    80987      +199     
     Branches      12002    12081       +79     
   =============================================
   - Hits          57953    25055    -32898     
   - Misses        18954    53781    +34827     
   + Partials       3881     2151     -1730     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `29.34% <16.38%> (-0.15%)` | :arrow_down: |
   | integration2 | `27.86% <11.23%> (-0.10%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `16.96% <16.96%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `12.50% <19.44%> (-37.03%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `61.07% <39.02%> (-28.89%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `82.23% <40.00%> (-13.98%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `46.66% <40.00%> (-13.34%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `53.48% <40.00%> (-24.78%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1060 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...a4b6d90](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] kishoreg commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

kishoreg commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982088481

I like the concept of the pushdown evaluator. Few thoughts/concerns
- Some methods added to projectionBlock seem to be specific to projectionBlock and is deviating from the block concept of chaining any blocks together i.e. the caller of the block does not understand the implementation of the underlying block. We have violated this in few places and we should fix that. It might be a good idea to enhance the Block interface to allow a pushdown evaluator.
- Conceptually, we are moving from a Column to Column + path but modelling the path as an evaluator. But the query layer is deciding the evaluator instead of the storage layer.
- The evaluator seems to be set at the JVM level but in practice, it's possible for each table to have a different evaluator and in fact, it's possible that each segment can have a different evaluator.

At a high level, the segment is the smallest unit of query processing in Pinot and most of the things can be decided independently at that scope - segmentDirectory, readers, writers, Blocks, indexes, encoding etc.

Having said that, I don't have a very good solution yet but I think we can come up with something after a couple of iterations. One thought is enhancing the concept (String Column) to a Column Object which has SingleColumn, NestedColumn( column + path) as implementations.

Overall, I think the Storage (Reader) should decide on the evaluator instead of the query layer.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759970147



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       I added these because I think the test setup is very good, it can actually be used to test cases on `jsonextractscalar` that weren't previously tested, and these columns are only used to test `jsonextractscalar`. Indeed, JSON literals did not work and still do not work with `BYTES` and `STRING` Does this alleviate your concern?

##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       I added these because I think the test setup is very good, it can actually be used to test cases on `jsonextractscalar` that weren't previously tested, and these columns are only used to test `jsonextractscalar`. Indeed, JSON literals did not work and still do not work with `BYTES` and `STRING`.
   
   Does this alleviate your concern?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762851291



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
##########
@@ -559,12 +744,21 @@ void readStringValuesMV(int[] docIds, int length, String[][] valuesBuffer) {
       }
     }
 
+    void readStringValuesMV(TransformEvaluator evaluator, int[] docIds, int length, String[][] valuesBuffer) {
+      evaluator.evaluateBlock(docIds, length, _reader, getReaderContext(), _dictionary, getSVDictIdsBuffer(),
+          valuesBuffer);
+    }
+
     public void readNumValuesMV(int[] docIds, int length, int[] numValuesBuffer) {
       for (int i = 0; i < length; i++) {
-        numValuesBuffer[i] = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, getReaderContext());
+        numValuesBuffer[i] = _reader.getDictIdMV(docIds[i], getSVDictIdsBuffer(), getReaderContext());

Review comment:
       good catch, this was an unintentional change made during the churn




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762972543



##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluator.java
##########
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import org.apache.pinot.segment.spi.evaluator.TransformEvaluator;
+
+/**
+ * Introduce an empty interface to allow it to be extended without
+ * affecting {@see TransformEvaluator}.
+ *
+ * This is an evolving SPI and subject to change.
+ */
+public interface JsonPathEvaluator extends TransformEvaluator {

Review comment:
       Can they stay here for now? They need to be implemented by a plugin which ideally would not depend on pinot-core.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-984548475


   @Jackie-Jiang to demonstrate an open mind, I've implemented your suggestion here: https://github.com/apache/pinot/pull/7820/commits/f75135887001394abb4097a9ddae486e5cea1c48
   
   I'm not convinced this is necessarily cleaner (`ProjectionBlock` is though) and I think it was better to keep the "tunnel" for evaluators separate from the regular path with caching. Writing the array copies from the enormous array maintained by `DataBlockCache` (because it doesn't have enough information to size the result properly) to the correctly sized arrays in `JsonExtractScalarFunction` made me want to cry ever so slightly. 
   
   Please pay attention to the changes to the cache key in `DataBlockCache` - JSONPath evaluations need to be discriminated by both the JSON Path itself and the default value, on top of the column name and type of the result. This is probably better than having large JSON documents lingering in the cache, but I'm struggling to see this cache as an unequivocal benefit for every possible transformation. 
   
   Could we avoid caching JSON path evaluations at the block level and revert the commit? If we find that we spend a lot of time on duplicate JSON path evaluations, could we think about avoiding the planning of duplicate evaluations instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mayankshriv commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

mayankshriv commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r755550432



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       Does it make sense to have a more generic evaluator abstraction (with JsonPathEvaluator as one of the impls) here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r755551967



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       I believe in the rule of 3: create a generic abstraction once you have three implementations.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5e1cfe) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `6.70%`.
   > The diff coverage is `37.97%`.
   
   > :exclamation: Current head a5e1cfe differs from pull request most recent head db04a8c. Consider uploading reports for the commit db04a8c to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.66%   64.96%   -6.71%     
   + Complexity     4088     4083       -5     
   ============================================
     Files          1579     1537      -42     
     Lines         80851    79512    -1339     
     Branches      12017    11924      -93     
   ============================================
   - Hits          57938    51651    -6287     
   - Misses        19017    24173    +5156     
   + Partials       3896     3688     -208     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.29% <37.97%> (-0.39%)` | :arrow_down: |
   | unittests2 | `14.53% <0.00%> (-0.11%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/core/common/Block.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vQmxvY2suamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `27.27% <0.00%> (-41.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `31.44% <31.44%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `22.49% <35.64%> (-27.04%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `88.81% <60.00%> (-7.40%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.71% <60.97%> (-4.25%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `63.15% <63.15%> (ø)` | |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [366 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...db04a8c](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r757700014



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -257,6 +313,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int[][] values for a single-valued column.

Review comment:
       No: the JSON column is single value, but the JsonPath produces a multi-valued result. If it’s confusing the verbiage can be expanded upon.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760286325



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       I have made a change to prevent changing this particular test here (which any kind of schema change always will, the way the test has been written): https://github.com/apache/pinot/pull/7820/commits/220c16c6683240cb840dc7a1891a947e38ae5164
   
   However, my preference is to drop that commit and make the change above to make the test stable to schema changes. The schema change is beneficial, because it tests storage configurations (e.g. RAW JSON) which wasn't previously tested.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] kishoreg commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

kishoreg commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760388945



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       Unrelated to this PR. @amrishlal when did we make the decision that json_extract should work only on JSON datatype? this is a backward-incompatible change. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760286325



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       I have made a change to prevent changing this particular test here (which any kind of schema change always will, the way the test has been written): https://github.com/apache/pinot/pull/7820/commits/b71cd8df997bd52b3e40abdf6ebcae103b44a81e
   
   However, my preference is to drop that commit and make the change above to make the test stable to schema changes. The schema change is beneficial, because it tests storage configurations (e.g. RAW JSON) which wasn't previously tested.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985428914


   > For now, we might just add some read methods into the ProjectionBlock so that it can be used by the TransformFunction? Adding these methods to the Block interface might not match our end goal.
   
   This is what the PR originally did, I did not modify the `Block` interface and added methods to the concrete `ProjectionBlock`, but it was requested above to ensure substitutability of different block types. It feels like I am violating an aspirational design principle with this PR, but as far as I can tell, it has been violated for a long time. A lot of good things will follow on from the ability to tunnel down to the storage, so I would like to find a way to progress here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fa0219f) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `6.53%`.
   > The diff coverage is `39.92%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.66%   65.12%   -6.54%     
   + Complexity     4088     4084       -4     
   ============================================
     Files          1579     1538      -41     
     Lines         80851    79979     -872     
     Branches      12017    12034      +17     
   ============================================
   - Hits          57938    52089    -5849     
   - Misses        19017    24175    +5158     
   + Partials       3896     3715     -181     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.52% <39.92%> (-0.17%)` | :arrow_down: |
   | unittests2 | `14.44% <0.00%> (-0.20%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `31.44% <31.44%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `22.49% <35.64%> (-27.04%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `88.81% <60.00%> (-7.40%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.71% <60.97%> (-4.25%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `61.11% <61.11%> (ø)` | |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [444 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...fa0219f](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

siddharthteotia commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985933673


   > I'll update the Javadoc but I told @siddharthteotia that this would not be merged until he's had a chance to review it.
   
   @richardstartin, thanks for waiting. I pulled the changes locally and understood them. Had a few minor questions which I met with @Jackie-Jiang to clarify. LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762105653



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -257,6 +313,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int[][] values for a single-valued column.

Review comment:
       Just circling back here - this is ambiguous now the evaluator concept is more general than JSON, so I will remove the comment about the arity of the column.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759974790



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       I'm not sure I agree about that, I don't think that this test was stable w.r.t. extension of the test setup. 
   
   In order to add columns to the test setup (e.g. to test a RAW JSON column, which was not previously tested, and is now) `expecteds1` would always need to be extended to include every column added. So there is a choice: add the values to `expected1`, duplicate the whole test _setup_ to add the new columns for a new test, or control what is projected to keep this test stable. Since this is a test for selection and not projection, I opted to do the latter. 
   
   So do you want me to duplicate the test setup so I can add tests for querying a RAW JSON column, or change the query back to `select *` and add all new columns to `expected1`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760286325



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       I have made a change to prevent changing this particular test here (which any kind of schema change always will, the way the test has been written): https://github.com/apache/pinot/pull/7820/commits/76f703540e0f617799d7bce459c4c1fce3da5e85
   
   However, my preference is to drop that commit and make the change above to make the test stable to schema changes. The schema change is beneficial, because it tests storage configurations (e.g. RAW JSON) which wasn't previously tested.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] amrishlal commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

amrishlal commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760630376



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       @richardstartin Yes, it resolves the issue. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985674938


   > Curious, do you have some benchmark number to show how much perf improvement?
   
   Performance improvement isn't the primary motivation but to provide a plugin mechanism which allows query logic to depend on the available storage. Before merging I would, of course, provide evidence that this does not _regress_ performance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2489695) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `43.93%`.
   > The diff coverage is `19.55%`.
   
   > :exclamation: Current head 2489695 differs from pull request most recent head bcb55a2. Consider uploading reports for the commit bcb55a2 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   27.79%   -43.94%     
   =============================================
     Files          1579     1572        -7     
     Lines         80788    81069      +281     
     Branches      12002    12072       +70     
   =============================================
   - Hits          57953    22537    -35416     
   - Misses        18954    56507    +37553     
   + Partials       3881     2025     -1856     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `27.79% <19.55%> (-0.17%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [...ceselector/StrictReplicaGroupInstanceSelector.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL1N0cmljdFJlcGxpY2FHcm91cEluc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `62.12% <ø> (-27.28%)` | :arrow_down: |
   | [...oker/routing/segmentpruner/EmptySegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL0VtcHR5U2VnbWVudFBydW5lci5qYXZh) | `74.00% <ø> (-26.00%)` | :arrow_down: |
   | [.../routing/segmentpruner/PartitionSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1BhcnRpdGlvblNlZ21lbnRQcnVuZXIuamF2YQ==) | `0.00% <0.00%> (-68.22%)` | :arrow_down: |
   | [...roker/routing/segmentpruner/TimeSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1RpbWVTZWdtZW50UHJ1bmVyLmphdmE=) | `0.00% <0.00%> (-80.96%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [.../apache/pinot/spi/config/table/IndexingConfig.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0luZGV4aW5nQ29uZmlnLmphdmE=) | `0.00% <ø> (-92.21%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `7.94% <7.94%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `7.77% <10.18%> (-41.76%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `56.78% <19.51%> (-33.18%)` | :arrow_down: |
   | ... and [1132 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...bcb55a2](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762942133



##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluators.java
##########
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import com.google.common.base.Preconditions;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Allows registration of a custom {@see JsonPathEvaluator} which can handle custom storage
+ * functionality also present in a plugin. A default evaluator which can handle all default
+ * storage types will be provided to delegate to when standard storage types are encountered.
+ *
+ * This is an evolving SPI and subject to change.
+ */
+public final class JsonPathEvaluators {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JsonPathEvaluators.class);
+
+  private static final AtomicReferenceFieldUpdater<JsonPathEvaluators, JsonPathEvaluatorProvider> UPDATER =
+      AtomicReferenceFieldUpdater.newUpdater(JsonPathEvaluators.class, JsonPathEvaluatorProvider.class, "_provider");
+  private static final JsonPathEvaluators INSTANCE = new JsonPathEvaluators();
+  private static final DefaultProvider DEFAULT_PROVIDER = new DefaultProvider();
+  private volatile JsonPathEvaluatorProvider _provider;
+
+  /**
+   * Registration point to override how JSON paths are evaluated. This should be used
+   * when a Pinot plugin has special storage capabilities. For instance, imagine a
+   * plugin with a raw forward index which stores JSON in a binary format which
+   * pinot-core is unaware of and cannot evaluate JSON paths against (pinot-core only
+   * understands true JSON documents). Whenever JSON paths are evaluated against this
+   * custom storage, different storage access operations may be required, and the provided
+   * {@see JsonPathEvaluator} can inspect the provided {@see ForwardIndexReader} to
+   * determine whether it is the custom implementation and evaluate the JSON path against
+   * the binary JSON managed by the custom reader. If it is not the custom implementation,
+   * then the evaluation should be delegated to the provided delegate.
+   *
+   * This prevents the interface {@see ForwardIndexReader} from needing to be able to model
+   * any plugin storage format, which creates flexibility for the kinds of data structure
+   * plugins can employ.
+   *
+   * @param provider provides {@see JsonPathEvaluator}
+   * @return true if registration is successful, false otherwise
+   */
+  public static boolean registerProvider(JsonPathEvaluatorProvider provider) {
+    Preconditions.checkArgument(provider != null, "");
+    if (!UPDATER.compareAndSet(INSTANCE, null, provider)) {
+      LOGGER.warn("failed to register {} - {} already registered", provider, INSTANCE._provider);
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * pinot-core must construct {@see JsonPathEvaluator} via this method to ensure it uses
+   * the registered implementation. Using the registered implementation allows pinot-core
+   * to evaluate JSON paths against data structures it doesn't understand or model.
+   * @param jsonPath the JSON path
+   * @param defaultValue the default value
+   * @return a JSON path evaluator which must understand all possible storage representations of JSON.
+   */
+  public static JsonPathEvaluator create(String jsonPath, Object defaultValue) {
+    // plugins compose and delegate to the default implementation.
+    JsonPathEvaluator defaultEvaluator = DEFAULT_PROVIDER.create(jsonPath, defaultValue);
+    return Holder.PROVIDER.create(defaultEvaluator, jsonPath, defaultValue);
+  }
+
+  /**
+   * Storing the registered evaluator in this holder and initialising it during
+   * the class load gives the best of both worlds: plugins have until the first
+   * JSON path evaluation to register an evaluator via
+   * {@see JsonPathEvaluators#registerProvider}, but once this class is loaded,
+   * the provider is constant and calls may be optimise aggressively by the JVM
+   * in ways which are impossible with a volatile reference.
+   */
+  private static final class Holder {
+    static final JsonPathEvaluatorProvider PROVIDER;
+
+    static {
+      JsonPathEvaluatorProvider provider = JsonPathEvaluators.INSTANCE._provider;
+      if (provider == null) {
+        provider = DEFAULT_PROVIDER;
+        if (!UPDATER.compareAndSet(INSTANCE, null, provider)) {
+          provider = JsonPathEvaluators.INSTANCE._provider;
+        }
+      }
+      PROVIDER = provider;
+    }
+  }
+
+  private static class DefaultProvider implements JsonPathEvaluatorProvider {
+
+    // default implementation uses MethodHandles to avoid pulling lots of implementation details into the SPI layer
+
+    private static final MethodHandle FACTORY;
+
+    static {
+      String className = "org.apache.pinot.core.common.evaluators.DefaultJsonPathEvaluator";
+      MethodHandle factory = null;
+      try {
+        Class<?> clazz = Class.forName(className, false, JsonPathEvaluators.class.getClassLoader());
+        factory = MethodHandles.publicLookup()
+            .findStatic(clazz, "create", MethodType.methodType(JsonPathEvaluator.class, String.class, Object.class));
+      } catch (Throwable implausible) {
+        LOGGER.error("could not construct MethodHandle for {}", className,
+            implausible);
+      }
+      FACTORY = factory;
+    }
+
+    public JsonPathEvaluator create(String jsonPath, Object defaultValue) {
+      return create(null, jsonPath, defaultValue);
+    }
+
+    @Override
+    public JsonPathEvaluator create(JsonPathEvaluator delegate, String jsonPath, Object defaultValue) {

Review comment:
       The idea here is there is a chain of responsibility. There is the standard way of evaluating JSON paths (the result of the refactoring here), but a plugin could be registered which adds special storage as well as an evaluator which can interpret that storage. There could be a mix of segments using the default storage and the plugin storage. On a case by case basis, the registered plugin either handles the encountered storage or delegates if it doesn't know how. 
   
   I'm not sure what I was thinking when I wrote this particular line though - `DefaultJsonPathEvaluator` is the end of the chain and will never delegate, so its delegate will always be null.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fa0219f) into [master](https://codecov.io/gh/apache/pinot/commit/e572ea02dd942e3eeaad71233b91d8d003f5f2d4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e572ea0) will **decrease** coverage by `1.39%`.
   > The diff coverage is `39.92%`.
   
   > :exclamation: Current head fa0219f differs from pull request most recent head a89359a. Consider uploading reports for the commit a89359a to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.65%   70.26%   -1.40%     
   - Complexity     4080     4084       +4     
   ============================================
     Files          1581     1583       +2     
     Lines         81350    81849     +499     
     Branches      12128    12237     +109     
   ============================================
   - Hits          58293    57509     -784     
   - Misses        19117    20397    +1280     
   - Partials       3940     3943       +3     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `27.69% <11.32%> (-0.11%)` | :arrow_down: |
   | unittests1 | `68.52% <39.92%> (-0.29%)` | :arrow_down: |
   | unittests2 | `14.44% <0.00%> (-0.08%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `31.44% <31.44%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `22.49% <35.64%> (-27.04%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `91.44% <60.00%> (-4.77%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.71% <60.97%> (-4.25%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `61.11% <61.11%> (ø)` | |
   | [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...e/pinot/common/minion/MergeRollupTaskMetadata.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWluaW9uL01lcmdlUm9sbHVwVGFza01ldGFkYXRhLmphdmE=) | `0.00% <0.00%> (-94.74%)` | :arrow_down: |
   | ... and [105 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e572ea0...a89359a](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982140750

To illustrate the evaluator’s responsibility to understand the storage, the DefaultJsonPathEvaluator in this PR knows how to work with all storage capabilities for storing JSON currently present in Pinot (dictionarised string, bytes, raw string, bytes, JSON type column or not) but adding a storage plugin requires creating a new evaluator which understands the custom storage but delegating to a default evaluator for everything else, and registering it via the SPI.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982140750

To illustrate the evaluator’s responsibility to understand the storage, the DefaultJsonPathEvaluator in this PR knows how to work with all storage capabilities for storing JSON currently present in Pinot (dictionarised string, bytes, raw string, bytes, JSON type column or not) but adding a storage plugin means creating a new evaluator which understands the custom storage but delegating to a default evaluator for everything else, and registering it via the SPI.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759517406



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
##########
@@ -100,6 +101,19 @@ public void fetchIntValues(String column, int[] inDocIds, int length, int[] outV
     _columnValueReaderMap.get(column).readIntValues(inDocIds, length, outValues);
   }
 
+  /**
+   * Fetch the int values from a JSON column.

Review comment:
       This is a carry over from before generalising the interface. Will fix all of them.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-984389687


   @Jackie-Jiang the reason I bypassed the cache is because the cache would need a composite key to avoid different JSON path transformations sharing results spuriously. It’s not a tangential optimisation to get rid of a few buffers or array copies. Could this be handled differently without the cache by reordering the execution graph to deduplicate common subexpressions? From my perspective, it doesn’t feel like a huge departure to allow some expressions to bypass the cache, and I don’t think it should be necessary for all expressions to bypass the cache if some do.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977264186


   The try/catches can be removed if https://github.com/json-path/JsonPath/pull/767 is merged and released.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (76f7035) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `57.14%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.51%   -57.15%     
   + Complexity     4088       80     -4008     
   =============================================
     Files          1579     1537       -42     
     Lines         80851    79480     -1371     
     Branches      12017    11924       -93     
   =============================================
   - Hits          57938    11539    -46399     
   - Misses        19017    67090    +48073     
   + Partials       3896      851     -3045     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.51% <0.00%> (-0.12%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/core/common/Block.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vQmxvY2suamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `0.00% <0.00%> (-69.24%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1258 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...76f7035](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (862fee2) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `57.14%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.51%   -57.15%     
   + Complexity     4088       80     -4008     
   =============================================
     Files          1579     1537       -42     
     Lines         80851    79480     -1371     
     Branches      12017    11924       -93     
   =============================================
   - Hits          57938    11538    -46400     
   - Misses        19017    67089    +48072     
   + Partials       3896      853     -3043     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.51% <0.00%> (-0.12%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/core/common/Block.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vQmxvY2suamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `0.00% <0.00%> (-69.24%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1259 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...862fee2](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760286325



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       I have made a change to prevent changing this particular test here (which any kind of schema change always will, the way the test has been written): https://github.com/apache/pinot/pull/7820/commits/909cd9cf68abf180b24ded953a2e821b6f8c8dfb
   
   However, my preference is to drop that commit and make the change above to make the test stable to schema changes. The schema change is beneficial, because it tests storage configurations (e.g. RAW JSON) which wasn't previously tested.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] amrishlal commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

amrishlal commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760591685



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       @richardstartin This unit test file is only to test JSON path expressions on `JSON` column. If you are trying to validate JSON_EXTRACT_SCALAR function, I think a better place for it would be JsonDataTypeQueriesTest.java which has a lot of test cases for json_extract_scalar. I haven't looked through the PR in detail, so maybe I am missing something, but will look into more detail.
   
   @kishoreg Perhaps that was just my own personal impression at the time we were code reviewing JSON datatype. I suppose it should be ok to allow `JSON_EXTRACT_SCALAR` to work over `BYTE`, but then there is no guarantee that `BYTE` or `STRING` would contain JSON data (?), i.e any valid or invalid JSON data could be put into a `STRING` or `BYTE` column. If those byes or string are ingested into `JSON` datatype column, then we know for sure that it is valid JSON and can treat it as such during query processing, indexing, etc.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mayankshriv commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

mayankshriv commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r758704062



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       Yeah I see your point about the rule of 3. I think I put the comment in a less effective place, since this is not an interface implementation.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762856827



##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluators.java
##########
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import com.google.common.base.Preconditions;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Allows registration of a custom {@see JsonPathEvaluator} which can handle custom storage
+ * functionality also present in a plugin. A default evaluator which can handle all default
+ * storage types will be provided to delegate to when standard storage types are encountered.
+ *
+ * This is an evolving SPI and subject to change.
+ */
+public final class JsonPathEvaluators {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JsonPathEvaluators.class);
+
+  private static final AtomicReferenceFieldUpdater<JsonPathEvaluators, JsonPathEvaluatorProvider> UPDATER =
+      AtomicReferenceFieldUpdater.newUpdater(JsonPathEvaluators.class, JsonPathEvaluatorProvider.class, "_provider");
+  private static final JsonPathEvaluators INSTANCE = new JsonPathEvaluators();
+  private static final DefaultProvider DEFAULT_PROVIDER = new DefaultProvider();
+  private volatile JsonPathEvaluatorProvider _provider;
+
+  /**
+   * Registration point to override how JSON paths are evaluated. This should be used
+   * when a Pinot plugin has special storage capabilities. For instance, imagine a
+   * plugin with a raw forward index which stores JSON in a binary format which
+   * pinot-core is unaware of and cannot evaluate JSON paths against (pinot-core only
+   * understands true JSON documents). Whenever JSON paths are evaluated against this
+   * custom storage, different storage access operations may be required, and the provided
+   * {@see JsonPathEvaluator} can inspect the provided {@see ForwardIndexReader} to
+   * determine whether it is the custom implementation and evaluate the JSON path against
+   * the binary JSON managed by the custom reader. If it is not the custom implementation,
+   * then the evaluation should be delegated to the provided delegate.
+   *
+   * This prevents the interface {@see ForwardIndexReader} from needing to be able to model
+   * any plugin storage format, which creates flexibility for the kinds of data structure
+   * plugins can employ.
+   *
+   * @param provider provides {@see JsonPathEvaluator}
+   * @return true if registration is successful, false otherwise
+   */
+  public static boolean registerProvider(JsonPathEvaluatorProvider provider) {
+    Preconditions.checkArgument(provider != null, "");

Review comment:
       I think this is better for a couple of reasons
   1. double registration becomes obvious
   2. various runtime benefits of the provider being `final`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759519302



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/PushDownTransformFunction.java
##########
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.segment.spi.evaluator.TransformEvaluator;
+
+
+public interface PushDownTransformFunction {

Review comment:
       👍🏻 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982539768


   @kishoreg please see the updated Javadoc which provides more colour.
   
   @siddharthteotia you expressed some interest in this PR, would you mind taking a look please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang merged pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang merged pull request #7820:
URL: https://github.com/apache/pinot/pull/7820


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad5aa15) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `1.49%`.
   > The diff coverage is `33.02%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.73%   70.24%   -1.50%     
   - Complexity     4087     4089       +2     
   ============================================
     Files          1579     1581       +2     
     Lines         80788    81336     +548     
     Branches      12002    12096      +94     
   ============================================
   - Hits          57953    57134     -819     
   - Misses        18954    20304    +1350     
   - Partials       3881     3898      +17     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `27.88% <10.94%> (-0.08%)` | :arrow_down: |
   | unittests1 | `68.29% <33.02%> (-0.39%)` | :arrow_down: |
   | unittests2 | `14.50% <0.00%> (-0.08%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `23.20% <23.20%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `17.56% <28.70%> (-31.96%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `57.57% <57.57%> (ø)` | |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.35% <58.53%> (-4.61%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `91.44% <60.00%> (-4.77%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...e/pinot/common/minion/MergeRollupTaskMetadata.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWluaW9uL01lcmdlUm9sbHVwVGFza01ldGFkYXRhLmphdmE=) | `0.00% <0.00%> (-94.74%)` | :arrow_down: |
   | ... and [123 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...ad5aa15](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r755552508



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       There is no external impact of changing `JsonPathEvaluator` to `Evaluator` which could be a regular expression (or something else) later, it all stays in pinot-core.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r758722103



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       OK. I think what’s important about this API is the _shape_ of the result, and JsonPath evaluation will only ever produce a single vector of values with the same row indexing, in the same way that, say, a pushed down regular expression would. There are other obvious shapes to consider, such as an aggregator which produces a single value, or potentially something like a JSON UNNEST function which produces a matrix of values, typically with different row indexing. So any early abstraction for this needs to be able to model shape adequately. JsonPathEvaluator could be generalised to _TransformEvaluator_ because it preserves the row indexing.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-983918350


   @kishoreg Specifying in Javadoc that these are unstable APIs is an excellent compromise. I think it should be possible to evolve towards what you want. I must admit that I don't see it right now, but I'm trying hard to find a way both to implement the suggestion and am thinking about the benefits. I want to make it very easy to implement storage plugins (as an idea incubation mechanism) without needing community consensus on the shape of SPIs like `ForwardIndexReader` which will affect a lot of stakeholders. A delegation chain of `TransformEvaluator`s, marked as an unstable interface, feels like a good way to iterate without stalling for consensus or imposing churn on stakeholders.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mayankshriv commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

mayankshriv commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r758704062



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -120,6 +121,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int values for a single-valued column.
+   *
+   * @param column Column name
+   * @param evaluator JSON path evaluator
+   * @param buffer values to fill
+   */
+  public void fillValues(String column, JsonPathEvaluator evaluator, int[] buffer) {

Review comment:
       Yeah I see your point about the rule of 3, it is just that the `JsonPathEvaulator` argument seems a bit out of the place (and too special-cased). This is not really an api in an interface, so I guess it should be easier to change when we need to.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

klsince commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759502515



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
##########
@@ -100,6 +101,19 @@ public void fetchIntValues(String column, int[] inDocIds, int length, int[] outV
     _columnValueReaderMap.get(column).readIntValues(inDocIds, length, outValues);
   }
 
+  /**
+   * Fetch the int values from a JSON column.

Review comment:
       nit: is `JSON column` too specific for the method? 

##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/TransformEvaluator.java
##########
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator;
+
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+
+
+public interface TransformEvaluator {
+  /**
+   * Evaluate the JSON path and fill the value buffer

Review comment:
       nit: `JSON path` is too specific here?

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/PushDownTransformFunction.java
##########
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.segment.spi.evaluator.TransformEvaluator;
+
+
+public interface PushDownTransformFunction {

Review comment:
       nit: maybe add some comments about who can implement this interface?

##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluators.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import com.google.common.base.Preconditions;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Allows registration of a custom {@see JsonPathEvaluator} which can handle custom storage
+ * functionality also present in a plugin. A default evaluator which can handle all default
+ * storage types will be provided to delegate to when standard storage types are encountered.
+ */
+public final class JsonPathEvaluators {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JsonPathEvaluators.class);
+
+  private static final AtomicReferenceFieldUpdater<JsonPathEvaluators, JsonPathEvaluatorProvider> UPDATER =
+      AtomicReferenceFieldUpdater.newUpdater(JsonPathEvaluators.class, JsonPathEvaluatorProvider.class, "_provider");
+  private static final JsonPathEvaluators INSTANCE = new JsonPathEvaluators();
+  private static final DefaultProvider DEFAULT_PROVIDER = new DefaultProvider();
+  private volatile JsonPathEvaluatorProvider _provider;
+
+  /**
+   * Registration point to override how JSON paths are evaluated. This should be used
+   * when a Pinot plugin has special storage capabilities. For instance, imagine a
+   * plugin with a raw forward index which stores JSON in a binary format which
+   * pinot-core is unaware of and cannot evaluate JSON paths against (pinot-core only
+   * understands true JSON documents). Whenever JSON paths are evaluated against this
+   * custom storage, different storage access operations may be required, and the provided
+   * {@see JsonPathEvaluator} can inspect the provided {@see ForwardIndexReader} to
+   * determine whether it is the custom implementation and evaluate the JSON path against
+   * the binary JSON managed by the custom reader. If it is not the custom implementation,
+   * then the evaluation should be delegated to the provided delegate.
+   *
+   * This prevents the interface {@see ForwardIndexReader} from needing to be able to model
+   * any plugin storage format, which creates flexibility for the kinds of data structure
+   * plugins can employ.
+   *
+   * @param provider provides {@see JsonPathEvaluator}
+   * @return true if registration is successful, false otherwise
+   */
+  public static boolean registerProvider(JsonPathEvaluatorProvider provider) {
+    Preconditions.checkArgument(provider != null, "");
+    if (!UPDATER.compareAndSet(INSTANCE, null, provider)) {
+      LOGGER.warn("failed to register {} - {} already registered", provider, INSTANCE._provider);
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * pinot-core must construct {@see JsonPathEvaluator} via this method to ensure it uses
+   * the registered implementation. Using the registered implementation allows pinot-core
+   * to evaluate JSON paths against data structures it doesn't understand or model.
+   * @param jsonPath the JSON path
+   * @param defaultValue the default value
+   * @return a JSON path evaluator which must understand all possible storage representations of JSON.
+   */
+  public static JsonPathEvaluator create(String jsonPath, Object defaultValue) {
+    // plugins compose and delegate to the default implementation.
+    JsonPathEvaluator defaultEvaluator = DEFAULT_PROVIDER.create(jsonPath, defaultValue);
+    return Holder.PROVIDER.create(defaultEvaluator, jsonPath, defaultValue);
+  }
+
+  /**
+   * Storing the registered evaluator in this holder and initialising it during
+   * the class load gives the best of both worlds: plugins have until the first
+   * JSON path evaluation to register an evaluator via
+   * {@see JsonPathEvaluators#registerProvider}, but once this class is loaded,
+   * the provider is constant and calls may be optimise aggressively by the JVM
+   * in ways which are impossible with a volatile reference.
+   */
+  private static final class Holder {

Review comment:
       👍 

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/TransformBlock.java
##########
@@ -78,4 +79,54 @@ public BlockDocIdValueSet getBlockDocIdValueSet() {
   public BlockMetadata getMetadata() {
     throw new UnsupportedOperationException();
   }
+
+  @Override
+  public void pushDown(String column, TransformEvaluator evaluator, int[] buffer) {
+    _projectionBlock.pushDown(column, evaluator, buffer);

Review comment:
       any need to check `if (expression.getType() == ExpressionContext.Type.IDENTIFIER)` like L51 above? iiuc, we pushDown only when there is no transforms, i.e. a pass-through to projection block. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] amrishlal commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

amrishlal commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r759667776



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -177,40 +206,103 @@ private void checkresult(String query, Object[][] expecteds) {
       Assert.assertEquals(actual, expected);
     }
   }
+  @DataProvider
+  public static Object[][] allJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nativeJsonColumns() {
+    return new Object[][]{
+        {JSON_COLUMN},
+        {RAW_JSON_COLUMN},
+        {JSON_COLUMN_WITHOUT_INDEX},
+    };
+  }
+
+  @DataProvider
+  public static Object[][] nonNativeJsonColumns() {
+    // columns where we should be able to extract JSON with a function, but can't use all the literal features
+    return new Object[][]{
+        {RAW_BYTES_COLUMN},
+        {DICTIONARY_BYTES_COLUMN},
+        {RAW_STRING_COLUMN},
+        {DICTIONARY_STRING_COLUMN},
+    };
+  }
+
+  @Test(dataProvider = "nonNativeJsonColumns")
+  public void testExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(" + column + ", '$.name.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
+
+  @Test(dataProvider = "allJsonColumns")
+  public void testNestedExtractJsonField(String column) {
+    Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
+    checkresult("SELECT jsonextractscalar(jsonextractscalar(" + column
+        + ", '$.name', 'STRING'), '$.last', 'STRING') FROM testTable LIMIT 3", expecteds1);
+  }
 
   /** Test that a json path expression in SELECT list is properly converted to a JSON_EXTRACT_SCALAR function within
    * an AS function. */
-  @Test
-  public void testJsonSelect() {
+  @Test(dataProvider = "nativeJsonColumns")
+  public void testJsonSelect(String column) {
     // SELECT using a simple json path expression.
     Object[][] expecteds1 = {{"duck"}, {"mouse"}, {"duck"}};
-    checkresult("SELECT jsonColumn.name.last FROM testTable LIMIT 3", expecteds1);
+    checkresult("SELECT " + column + ".name.last FROM testTable LIMIT 3", expecteds1);
 
     Object[][] expecteds2 =
         {{"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"null"}, {"1"}};
-    checkresult("SELECT jsonColumn.data[0].e[2].z[0].i1 FROM testTable", expecteds2);
+    checkresult("SELECT " + column + ".data[0].e[2].z[0].i1 FROM testTable", expecteds2);
   }
 
   /** Test that a predicate comparing a json path expression with literal is properly converted into a JSON_MATCH
    * function. */
   @Test
   public void testJsonFilter() {
     // Comparing json path expression with a string value.
+    // note that only int, long, and string columns are projected so more kinds of json fields can be added without
+    // disrupting this test
     Object[][] expecteds1 =

Review comment:
       Please make sure that the existing test cases continue to work as is without modifying the expected results. To ensure compatiblity, I would suggest leaving the original test cases as is and adding new test cases if needed.

##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       JSON queries should work only on columns of JSON datatype, not on BYTES or STRING (I thought this was explicitly blocked already?). For backward compatibility, we still have JSON_EXTRACT_SCALAR and JSON_MATCH functions working on STRING column.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762855592



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java
##########
@@ -94,6 +95,7 @@ public void init(List<TransformFunction> arguments, Map<String, DataSource> data
         _defaultValue = dataType.convert(((LiteralTransformFunction) arguments.get(3)).getLiteral());
       }
       _resultMetadata = new TransformResultMetadata(dataType, isSingleValue, false);
+      _jsonPathEvaluator = JsonPathEvaluators.create(_jsonPathString, _defaultValue);

Review comment:
       This is intentionally like this. A plugin may not be implemented in terms of the JsonPath library, and even if it is, the compilation was cached across all segments in #7826




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762322462



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
##########
@@ -559,12 +744,21 @@ void readStringValuesMV(int[] docIds, int length, String[][] valuesBuffer) {
       }
     }
 
+    void readStringValuesMV(TransformEvaluator evaluator, int[] docIds, int length, String[][] valuesBuffer) {
+      evaluator.evaluateBlock(docIds, length, _reader, getReaderContext(), _dictionary, getSVDictIdsBuffer(),
+          valuesBuffer);
+    }
+
     public void readNumValuesMV(int[] docIds, int length, int[] numValuesBuffer) {
       for (int i = 0; i < length; i++) {
-        numValuesBuffer[i] = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, getReaderContext());
+        numValuesBuffer[i] = _reader.getDictIdMV(docIds[i], getSVDictIdsBuffer(), getReaderContext());

Review comment:
       This should not be changed

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/evaluators/DefaultJsonPathEvaluator.java
##########
@@ -0,0 +1,588 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.common.evaluators;
+
+import com.jayway.jsonpath.Configuration;
+import com.jayway.jsonpath.InvalidPathException;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.Option;
+import com.jayway.jsonpath.ParseContext;
+import com.jayway.jsonpath.spi.json.JacksonJsonProvider;
+import com.jayway.jsonpath.spi.mapper.JacksonMappingProvider;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import org.apache.pinot.common.function.JsonPathCache;
+import org.apache.pinot.segment.spi.evaluator.json.JsonPathEvaluator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.JsonUtils;
+
+
+public final class DefaultJsonPathEvaluator implements JsonPathEvaluator {
+
+  private static final ParseContext JSON_PARSER_CONTEXT = JsonPath.using(
+      new Configuration.ConfigurationBuilder().jsonProvider(new JacksonJsonProvider())
+          .mappingProvider(new JacksonMappingProvider()).options(Option.SUPPRESS_EXCEPTIONS).build());
+
+  private static final int[] EMPTY_INTS = new int[0];
+  private static final long[] EMPTY_LONGS = new long[0];
+  private static final float[] EMPTY_FLOATS = new float[0];
+  private static final double[] EMPTY_DOUBLES = new double[0];
+  private static final String[] EMPTY_STRINGS = new String[0];
+
+  public static JsonPathEvaluator create(String jsonPath, Object defaultValue) {

Review comment:
       (minor) Let's annotate `defaultValue` as `nullable` so that IDE can warn on potential null value access. Same for other methods with it as argument

##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluator.java
##########
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import org.apache.pinot.segment.spi.evaluator.TransformEvaluator;
+
+/**
+ * Introduce an empty interface to allow it to be extended without
+ * affecting {@see TransformEvaluator}.
+ *
+ * This is an evolving SPI and subject to change.
+ */
+public interface JsonPathEvaluator extends TransformEvaluator {

Review comment:
       These interfaces do not belong to the `segment-spi`, but `query-spi` which is not available yet. For now we can put them into the `pinot-core` which contains all the query related interfaces

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -69,4 +70,114 @@ public BlockDocIdValueSet getBlockDocIdValueSet() {
   public BlockMetadata getMetadata() {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * Pushes a {@see TransformEvaluator} which will produce an int value down
+   * to be evaluated against the column. This is an unstable API.
+   * @param column column to evaluate against
+   * @param evaluator the evaluator which produces values from the storage in the column
+   * @param buffer the buffer to write outputs into
+   */
+  public void pushDown(String column, TransformEvaluator evaluator, int[] buffer) {

Review comment:
       Suggest renaming it to `fillValues` which is more general, and can be used in the future for all value reads

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java
##########
@@ -94,6 +95,7 @@ public void init(List<TransformFunction> arguments, Map<String, DataSource> data
         _defaultValue = dataType.convert(((LiteralTransformFunction) arguments.get(3)).getLiteral());
       }
       _resultMetadata = new TransformResultMetadata(dataType, isSingleValue, false);
+      _jsonPathEvaluator = JsonPathEvaluators.create(_jsonPathString, _defaultValue);

Review comment:
       (optional) Consider directly passing the `JsonPath` into the evaluator so that we don't need to compile the json path in 2 places

##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluators.java
##########
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import com.google.common.base.Preconditions;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Allows registration of a custom {@see JsonPathEvaluator} which can handle custom storage
+ * functionality also present in a plugin. A default evaluator which can handle all default
+ * storage types will be provided to delegate to when standard storage types are encountered.
+ *
+ * This is an evolving SPI and subject to change.
+ */
+public final class JsonPathEvaluators {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JsonPathEvaluators.class);
+
+  private static final AtomicReferenceFieldUpdater<JsonPathEvaluators, JsonPathEvaluatorProvider> UPDATER =
+      AtomicReferenceFieldUpdater.newUpdater(JsonPathEvaluators.class, JsonPathEvaluatorProvider.class, "_provider");
+  private static final JsonPathEvaluators INSTANCE = new JsonPathEvaluators();
+  private static final DefaultProvider DEFAULT_PROVIDER = new DefaultProvider();
+  private volatile JsonPathEvaluatorProvider _provider;
+
+  /**
+   * Registration point to override how JSON paths are evaluated. This should be used
+   * when a Pinot plugin has special storage capabilities. For instance, imagine a
+   * plugin with a raw forward index which stores JSON in a binary format which
+   * pinot-core is unaware of and cannot evaluate JSON paths against (pinot-core only
+   * understands true JSON documents). Whenever JSON paths are evaluated against this
+   * custom storage, different storage access operations may be required, and the provided
+   * {@see JsonPathEvaluator} can inspect the provided {@see ForwardIndexReader} to
+   * determine whether it is the custom implementation and evaluate the JSON path against
+   * the binary JSON managed by the custom reader. If it is not the custom implementation,
+   * then the evaluation should be delegated to the provided delegate.
+   *
+   * This prevents the interface {@see ForwardIndexReader} from needing to be able to model
+   * any plugin storage format, which creates flexibility for the kinds of data structure
+   * plugins can employ.
+   *
+   * @param provider provides {@see JsonPathEvaluator}
+   * @return true if registration is successful, false otherwise
+   */
+  public static boolean registerProvider(JsonPathEvaluatorProvider provider) {
+    Preconditions.checkArgument(provider != null, "");
+    if (!UPDATER.compareAndSet(INSTANCE, null, provider)) {
+      LOGGER.warn("failed to register {} - {} already registered", provider, INSTANCE._provider);
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * pinot-core must construct {@see JsonPathEvaluator} via this method to ensure it uses
+   * the registered implementation. Using the registered implementation allows pinot-core
+   * to evaluate JSON paths against data structures it doesn't understand or model.
+   * @param jsonPath the JSON path
+   * @param defaultValue the default value
+   * @return a JSON path evaluator which must understand all possible storage representations of JSON.
+   */
+  public static JsonPathEvaluator create(String jsonPath, Object defaultValue) {
+    // plugins compose and delegate to the default implementation.
+    JsonPathEvaluator defaultEvaluator = DEFAULT_PROVIDER.create(jsonPath, defaultValue);
+    return Holder.PROVIDER.create(defaultEvaluator, jsonPath, defaultValue);
+  }
+
+  /**
+   * Storing the registered evaluator in this holder and initialising it during
+   * the class load gives the best of both worlds: plugins have until the first
+   * JSON path evaluation to register an evaluator via
+   * {@see JsonPathEvaluators#registerProvider}, but once this class is loaded,
+   * the provider is constant and calls may be optimise aggressively by the JVM
+   * in ways which are impossible with a volatile reference.
+   */
+  private static final class Holder {
+    static final JsonPathEvaluatorProvider PROVIDER;
+
+    static {
+      JsonPathEvaluatorProvider provider = JsonPathEvaluators.INSTANCE._provider;
+      if (provider == null) {
+        provider = DEFAULT_PROVIDER;
+        if (!UPDATER.compareAndSet(INSTANCE, null, provider)) {
+          provider = JsonPathEvaluators.INSTANCE._provider;
+        }
+      }
+      PROVIDER = provider;
+    }
+  }
+
+  private static class DefaultProvider implements JsonPathEvaluatorProvider {
+
+    // default implementation uses MethodHandles to avoid pulling lots of implementation details into the SPI layer
+
+    private static final MethodHandle FACTORY;
+
+    static {
+      String className = "org.apache.pinot.core.common.evaluators.DefaultJsonPathEvaluator";
+      MethodHandle factory = null;
+      try {
+        Class<?> clazz = Class.forName(className, false, JsonPathEvaluators.class.getClassLoader());
+        factory = MethodHandles.publicLookup()
+            .findStatic(clazz, "create", MethodType.methodType(JsonPathEvaluator.class, String.class, Object.class));
+      } catch (Throwable implausible) {
+        LOGGER.error("could not construct MethodHandle for {}", className,
+            implausible);
+      }
+      FACTORY = factory;
+    }
+
+    public JsonPathEvaluator create(String jsonPath, Object defaultValue) {
+      return create(null, jsonPath, defaultValue);
+    }
+
+    @Override
+    public JsonPathEvaluator create(JsonPathEvaluator delegate, String jsonPath, Object defaultValue) {

Review comment:
       Don't fully understand the logic here. What is the delegate evaluator used for?

##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/evaluator/json/JsonPathEvaluators.java
##########
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.evaluator.json;
+
+import com.google.common.base.Preconditions;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Allows registration of a custom {@see JsonPathEvaluator} which can handle custom storage
+ * functionality also present in a plugin. A default evaluator which can handle all default
+ * storage types will be provided to delegate to when standard storage types are encountered.
+ *
+ * This is an evolving SPI and subject to change.
+ */
+public final class JsonPathEvaluators {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JsonPathEvaluators.class);
+
+  private static final AtomicReferenceFieldUpdater<JsonPathEvaluators, JsonPathEvaluatorProvider> UPDATER =
+      AtomicReferenceFieldUpdater.newUpdater(JsonPathEvaluators.class, JsonPathEvaluatorProvider.class, "_provider");
+  private static final JsonPathEvaluators INSTANCE = new JsonPathEvaluators();
+  private static final DefaultProvider DEFAULT_PROVIDER = new DefaultProvider();
+  private volatile JsonPathEvaluatorProvider _provider;
+
+  /**
+   * Registration point to override how JSON paths are evaluated. This should be used
+   * when a Pinot plugin has special storage capabilities. For instance, imagine a
+   * plugin with a raw forward index which stores JSON in a binary format which
+   * pinot-core is unaware of and cannot evaluate JSON paths against (pinot-core only
+   * understands true JSON documents). Whenever JSON paths are evaluated against this
+   * custom storage, different storage access operations may be required, and the provided
+   * {@see JsonPathEvaluator} can inspect the provided {@see ForwardIndexReader} to
+   * determine whether it is the custom implementation and evaluate the JSON path against
+   * the binary JSON managed by the custom reader. If it is not the custom implementation,
+   * then the evaluation should be delegated to the provided delegate.
+   *
+   * This prevents the interface {@see ForwardIndexReader} from needing to be able to model
+   * any plugin storage format, which creates flexibility for the kinds of data structure
+   * plugins can employ.
+   *
+   * @param provider provides {@see JsonPathEvaluator}
+   * @return true if registration is successful, false otherwise
+   */
+  public static boolean registerProvider(JsonPathEvaluatorProvider provider) {
+    Preconditions.checkArgument(provider != null, "");

Review comment:
       Does it work if we simply put the `provider` as a `private static` field and set it during the instance startup? Several classes handle the plugin that way

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/PushDownTransformFunction.java
##########
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.segment.spi.evaluator.TransformEvaluator;
+
+
+public interface PushDownTransformFunction {

Review comment:
       (optional) Should we make this extend the `TransformFunction` interface, or do you see this independent of the regular `TransformFunction`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a390f09) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `6.57%`.
   > The diff coverage is `40.26%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.66%   65.08%   -6.58%     
   + Complexity     4088     4075      -13     
   ============================================
     Files          1579     1538      -41     
     Lines         80851    79981     -870     
     Branches      12017    12035      +18     
   ============================================
   - Hits          57938    52058    -5880     
   - Misses        19017    24194    +5177     
   + Partials       3896     3729     -167     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.44% <40.26%> (-0.24%)` | :arrow_down: |
   | unittests2 | `14.45% <0.00%> (-0.18%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `31.44% <31.44%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `22.49% <35.64%> (-27.04%)` | :arrow_down: |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `88.81% <60.00%> (-7.40%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.71% <61.90%> (-4.25%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `63.15% <63.15%> (ø)` | |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [443 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...a390f09](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a390f09) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `57.20%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.45%   -57.21%     
   + Complexity     4088       80     -4008     
   =============================================
     Files          1579     1538       -41     
     Lines         80851    79981      -870     
     Branches      12017    12035       +18     
   =============================================
   - Hits          57938    11563    -46375     
   - Misses        19017    67559    +48542     
   + Partials       3896      859     -3037     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.45% <0.00%> (-0.18%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1269 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...a390f09](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bcb55a2) into [master](https://codecov.io/gh/apache/pinot/commit/cce493214ecc5727ed2748d0d260fa7ba9e304e7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cce4932) will **decrease** coverage by `57.23%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.73%   14.50%   -57.24%     
   + Complexity     4087       80     -4007     
   =============================================
     Files          1579     1536       -43     
     Lines         80788    79547     -1241     
     Branches      12002    11907       -95     
   =============================================
   - Hits          57953    11535    -46418     
   - Misses        18954    67164    +48210     
   + Partials       3881      848     -3033     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.50% <0.00%> (-0.09%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1263 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cce4932...bcb55a2](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] kkrugler commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

kkrugler commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r757696416



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java
##########
@@ -257,6 +313,17 @@ public int getNumDocs() {
     return intValues;
   }
 
+  /**
+   * Get the int[][] values for a single-valued column.

Review comment:
       Isn't this getting values for a multi-valued column? Same comment for Javadocs below.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e362ace) into [master](https://codecov.io/gh/apache/pinot/commit/eff535134290a61b52a68a813a6022519d7ad1e0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff5351) will **decrease** coverage by `6.63%`.
   > The diff coverage is `39.57%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7820      +/-   ##
   ============================================
   - Coverage     71.66%   65.02%   -6.64%     
   - Complexity     4086     4087       +1     
   ============================================
     Files          1579     1536      -43     
     Lines         80839    79465    -1374     
     Branches      12010    11916      -94     
   ============================================
   - Hits          57930    51671    -6259     
   - Misses        19012    24107    +5095     
   + Partials       3897     3687     -210     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.35% <39.57%> (-0.33%)` | :arrow_down: |
   | unittests2 | `14.55% <0.00%> (-0.04%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `31.44% <31.44%> (ø)` | |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `22.49% <35.64%> (-27.04%)` | :arrow_down: |
   | [...che/pinot/segment/spi/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2pzb24vSnNvblBhdGhFdmFsdWF0b3JzLmphdmE=) | `57.57% <57.57%> (ø)` | |
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `88.81% <60.00%> (-8.16%)` | :arrow_down: |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <60.00%> (ø)` | |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `69.76% <60.00%> (-8.50%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `85.71% <60.97%> (-4.25%)` | :arrow_down: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [370 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff5351...e362ace](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r760433750



##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/JsonPathQueriesTest.java
##########
@@ -55,17 +60,37 @@
   private static final String LONG_COLUMN = "longColumn";
   private static final String STRING_COLUMN = "stringColumn";
   private static final String JSON_COLUMN = "jsonColumn";
+  private static final String RAW_JSON_COLUMN = "rawJsonColumn";
+  private static final String RAW_BYTES_COLUMN = "rawBytesColumn";
+  private static final String DICTIONARY_BYTES_COLUMN = "dictionaryBytesColumn";
+  private static final String RAW_STRING_COLUMN = "rawStringColumn";
+  private static final String DICTIONARY_STRING_COLUMN = "dictionaryStringColumn";
   private static final String JSON_COLUMN_WITHOUT_INDEX = "jsonColumnWithoutIndex";
 
   private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(RAW_TABLE_NAME)
       .addSingleValueDimension(INT_COLUMN, FieldSpec.DataType.INT)
       .addSingleValueDimension(LONG_COLUMN, FieldSpec.DataType.LONG)
       .addSingleValueDimension(STRING_COLUMN, FieldSpec.DataType.STRING)
       .addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_JSON_COLUMN, FieldSpec.DataType.JSON)
+      .addSingleValueDimension(RAW_BYTES_COLUMN, FieldSpec.DataType.BYTES)

Review comment:
       @kishoreg I don't think that's what @amrishlal was saying, this test focuses on jsonpath literals like `select * from table where foo.bar = 0` and these only work on JSON columns, so adding these other columns here so `jsonextractscalar` can be tested using the thorough set up this tests provides doesn't feel appropriate.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

codecov-commenter edited a comment on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7820](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fa0219f) into [master](https://codecov.io/gh/apache/pinot/commit/dfb7feef549f25cc907171125d7f77ecafaf2094?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dfb7fee) will **decrease** coverage by `57.21%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7820/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7820       +/-   ##
   =============================================
   - Coverage     71.66%   14.44%   -57.22%     
   + Complexity     4088       80     -4008     
   =============================================
     Files          1579     1538       -41     
     Lines         80851    79979      -872     
     Branches      12017    12034       +17     
   =============================================
   - Hits          57938    11551    -46387     
   - Misses        19017    67575    +48558     
   + Partials       3896      853     -3043     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.44% <0.00%> (-0.20%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
   | [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
   | [...re/common/evaluators/DefaultJsonPathEvaluator.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZXZhbHVhdG9ycy9EZWZhdWx0SnNvblBhdGhFdmFsdWF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
   | [...ransform/function/IdentifierTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSWRlbnRpZmllclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-78.27%)` | :arrow_down: |
   | [...m/function/JsonExtractScalarTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-49.53%)` | :arrow_down: |
   | [...segment/spi/evaluator/json/JsonPathEvaluators.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2V2YWx1YXRvci9qc29uL0pzb25QYXRoRXZhbHVhdG9ycy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1269 more](https://codecov.io/gh/apache/pinot/pull/7820/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [dfb7fee...fa0219f](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982559015

> Some methods added to projectionBlock seem to be specific to projectionBlock and is deviating from the block concept of chaining any blocks together i.e. the caller of the block does not understand the implementation of the underlying block. We have violated this in few places and we should fix that. It might be a good idea to enhance the Block interface to allow a pushdown evaluator.

@kishoreg this has been deviated from significantly already, and in many `Block` implementations there would be no sensible implementation but to throw `UnsupportedOperationException` which breaks LSP anyway.

I also want to call out that there appears to be an element of complexity which you may have missed: `TransformFunction` isn't guaranteed to operate on directly on storage, it may also operate on the output of some other JSON function (I added a test for this) - when that happens, there needs to be an implementation which can operate on values produced by another `TransformFunction`. This (to push down or evaluate directly) is a decision which can only be made in the query layer. I'm not sure if you didn't mention this because it makes sense already or not.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] kishoreg commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

kishoreg commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985676711


   @Jackie-Jiang let's try to get to the final version in a few iterations. I think it's better to keep the changes restricted to projection block for now. Let's get this to work and we can try to revisit the block interface in a different PR.
   
   I am +1 on this PR assuming we will mark the methods in projectionblock as evolving


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] yupeng9 commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

yupeng9 commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985672438


   Curious, do you have some benchmark number to show how much perf improvement?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

richardstartin commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985680307


   I'll update the Javadoc but I told @siddharthteotia that this would not be merged until he's had a chance to review it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

Posted by GitBox <gi...@apache.org>.

siddharthteotia commented on a change in pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#discussion_r762361654



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -69,4 +70,114 @@ public BlockDocIdValueSet getBlockDocIdValueSet() {
   public BlockMetadata getMetadata() {
     throw new UnsupportedOperationException();
   }
+
+  /**
+   * Pushes a {@see TransformEvaluator} which will produce an int value down
+   * to be evaluated against the column. This is an unstable API.
+   * @param column column to evaluate against
+   * @param evaluator the evaluator which produces values from the storage in the column
+   * @param buffer the buffer to write outputs into
+   */
+  public void pushDown(String column, TransformEvaluator evaluator, int[] buffer) {

Review comment:
       +1




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org