You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shenyu0127 (via GitHub)" <gi...@apache.org> on 2023/09/08 01:28:52 UTC

[GitHub] [pinot] shenyu0127 opened a new pull request, #11546: Fix AND and OR transform functions.

shenyu0127 opened a new pull request, #11546:
URL: https://github.com/apache/pinot/pull/11546

   The correct behavior is this: https://prestodb.io/docs/current/functions/logical.html.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319445980


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Even better, we could use the roaring bitmap iterator, which is going to be faster than sequential calls to `contains`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] xiangfu0 merged pull request #11546: Fix AND and OR transform functions.

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #11546:
URL: https://github.com/apache/pinot/pull/11546


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] shenyu0127 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1320682323


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +45,13 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  private boolean isFalse(int i) {

Review Comment:
   Removed the function.
   
   Had this function because another reviewer suggested that: https://github.com/apache/pinot/pull/11546#discussion_r1319452061.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11546: Fix AND and OR transform functions.

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11546:
URL: https://github.com/apache/pinot/pull/11546#issuecomment-1711004174

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11546](https://app.codecov.io/gh/apache/pinot/pull/11546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (23c2f3f) into [master](https://app.codecov.io/gh/apache/pinot/commit/6208e7cb467dd98216bcceafd14c0feb7b3bba6a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6208e7c) will **decrease** coverage by `48.58%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #11546       +/-   ##
   =============================================
   - Coverage     63.05%   14.48%   -48.58%     
   + Complexity     1109      201      -908     
   =============================================
     Files          2320     2320               
     Lines        124667   124707       +40     
     Branches      19033    19049       +16     
   =============================================
   - Hits          78614    18065    -60549     
   - Misses        40458   105113    +64655     
   + Partials       5595     1529     -4066     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.43%)` | :arrow_down: |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.58%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-48.57%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11546/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.48% <0.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...ansform/function/AndOperatorTransformFunction.java](https://app.codecov.io/gh/apache/pinot/pull/11546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vQW5kT3BlcmF0b3JUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ransform/function/OrOperatorTransformFunction.java](https://app.codecov.io/gh/apache/pinot/pull/11546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vT3JPcGVyYXRvclRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   
   ... and [1493 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11546/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] shenyu0127 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319445158


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {
+      boolean isFalse = false;
+      for (int i = 0; i < numArguments; i++) {
+        if ((nullBitmaps[i] == null || !nullBitmaps[i].contains(docId)) && intValuesSVs[i][docId] == 0) {

Review Comment:
   For AND operator, if any of the argument is false, then the result is false.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Done.
   
   You are right. Thank you for the catch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319445980


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Even better, we could use the roaring bitmap iterator, which is going to be faster than sequential calls to `contains`



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Even better, we could use the roaring bitmap iterator, which is going to be faster than sequential calls to contains



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] shenyu0127 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1320016706


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Done. Also moved the implementation to the base class.
   
   Good suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319438573


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];

Review Comment:
   I think this is incorrect from performance perspective. This sentence instantiates the whole matrix (it allocates an array of `numArguments` elements pointing to a new array of `numDocs` ints). Few lines later elements on `intValuesSVs` are changed to `_arguments.get(i).transformToIntValuesSV(valueBlock);`. Therefore we are creating the internal arrays for no reason.
   
   What you should do here is `new int[numArguments][];`
   
   You can verify that in jshell:
   
   ```
   ╰─➤  jshell                                                                                                                                                                         microk8s
   |  Welcome to JShell -- Version 11.0.19
   |  For an introduction type: /help intro
   
   jshell> var a = new int[2][3];
   a ==> int[2][] { int[3] { 0, 0, 0 }, int[3] { 0, 0, 0 } }
   
   jshell> a[0]
   $2 ==> int[3] { 0, 0, 0 }
   
   jshell> var b = new int[2][];
   b ==> int[2][] { null, null }
   
   jshell> b[0]
   $4 ==> null
   ```



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];

Review Comment:
   I think this is incorrect from performance perspective. This sentence instantiates the whole matrix (it allocates an array of `numArguments` elements pointing to a new array of `numDocs` ints). Few lines later elements on `intValuesSVs` are changed to `_arguments.get(i).transformToIntValuesSV(valueBlock);`. Therefore we are creating the internal arrays for no reason.
   
   What you should do here is `new int[numArguments][];`
   
   You can verify that in jshell:
   
   ```
   ╰─➤  jshell                                                                                                                                                                         microk8s
   |  Welcome to JShell -- Version 11.0.19
   |  For an introduction type: /help intro
   
   jshell> var a = new int[2][3];
   a ==> int[2][] { int[3] { 0, 0, 0 }, int[3] { 0, 0, 0 } }
   
   jshell> a[0]
   $2 ==> int[3] { 0, 0, 0 }
   
   jshell> var b = new int[2][];
   b ==> int[2][] { null, null }
   
   jshell> b[0]
   $4 ==> null
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319452061


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {
+      boolean isFalse = false;
+      for (int i = 0; i < numArguments; i++) {
+        if ((nullBitmaps[i] == null || !nullBitmaps[i].contains(docId)) && intValuesSVs[i][docId] == 0) {

Review Comment:
   Oh! and the int 0 is the value we use as false. That makes sense. Maybe the reason is that it is early morning here and I'm not 100% woken up, but it looks confusing to me. I would prefer to see something like `BooleanUtils.isFalse(intValuesSVs[i])`, but that method requires an Object, so it would imply to do boxing.
   
   We can create a `public static boolean isFalseInternalValue(int value)`, but I leave that decision to you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319442926


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {
+      boolean isFalse = false;
+      for (int i = 0; i < numArguments; i++) {
+        if ((nullBitmaps[i] == null || !nullBitmaps[i].contains(docId)) && intValuesSVs[i][docId] == 0) {

Review Comment:
   I don't understand this predicate. Specifically I don't understand why `!nullBitmaps[i].contains(docId)) && intValuesSVs[i][docId] == 0` implies that we don't need to check for the other docIds or arguments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319447515


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Even better, we could use the roaring bitmap iterator, which is going to be faster than sequential calls to contains
   
   (sorry for the spam, I deleted the original comment by mistake)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] shenyu0127 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319450751


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Please ignore my previous comment. It was replied to a wrong comment.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];

Review Comment:
   Done.
   
   You are right. Thank you for the catch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1320681554


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +45,13 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  private boolean isFalse(int i) {

Review Comment:
   why need this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] gortiz commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1319444154


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   I'm not sure if it is possible, but performance wise, I think it would be better to iterate over the arguments first in order to always use the same nullBitmap and therefore reduce the memory preasure.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   I'm not sure if it is possible, but performance wise, I think it would be better to iterate over the arguments first in order to always use the same nullBitmap and therefore reduce the memory pressure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] shenyu0127 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1320016706


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {

Review Comment:
   Done.
   
   Good suggestion!



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +48,37 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  @Nullable
+  @Override
+  public RoaringBitmap getNullBitmap(ValueBlock valueBlock) {
+    int numDocs = valueBlock.getNumDocs();
+    int numArguments = _arguments.size();
+    int[][] intValuesSVs = new int[numArguments][numDocs];
+    RoaringBitmap[] nullBitmaps = new RoaringBitmap[numArguments];
+    RoaringBitmap nullBitmap = new RoaringBitmap();
+    for (int i = 0; i < numArguments; i++) {
+      intValuesSVs[i] = _arguments.get(i).transformToIntValuesSV(valueBlock);
+      nullBitmaps[i] = _arguments.get(i).getNullBitmap(valueBlock);
+    }
+    for (int docId = 0; docId < numDocs; docId++) {
+      boolean isFalse = false;
+      for (int i = 0; i < numArguments; i++) {
+        if ((nullBitmaps[i] == null || !nullBitmaps[i].contains(docId)) && intValuesSVs[i][docId] == 0) {

Review Comment:
   Done.
   
   (If the code confused you, it means the code is not easy to read, and I should change it.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11546: Fix AND and OR transform functions.

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #11546:
URL: https://github.com/apache/pinot/pull/11546#discussion_r1320691185


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/AndOperatorTransformFunction.java:
##########
@@ -45,4 +45,13 @@ int getLogicalFuncResult(int left, int right) {
     }
     return 0;
   }
+
+  private boolean isFalse(int i) {

Review Comment:
   Ack, I don't think we need it here until one more reference is added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org