[GitHub] [carbondata] QiangCai commented on a change in pull request #3771: [WIP] pushdown array_contains filter to carbon

[GitBox <gi...@apache.org>]

QiangCai commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r429538025



##########
File path: core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
##########
@@ -67,4 +67,18 @@ private DimensionColumnPage getDecodedDimensionPage(DimensionColumnPage[][] dime
     }
     return dimensionColumnPages[columnIndex][pageNumber];
   }
+
+  /**
+   * Method will copy the block chunk holder data and return the cloned value.
+   * This method is also used by child.
+   */
+  protected byte[] copyBlockDataChunkWithoutClone(DimensionRawColumnChunk[] rawColumnChunks,
+      DimensionColumnPage[][] dimensionColumnPages, int rowNumber, int pageNumber) {
+    byte[] data =
+        getDecodedDimensionPage(dimensionColumnPages, rawColumnChunks[columnIndex], pageNumber)

Review comment:
       how about to cache the page, it will not require to decode for each row again.

##########
File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
##########
@@ -222,49 +224,90 @@ public BitSetGroup applyFilter(RawBlockletColumnChunks rawBlockletColumnChunks,
       }
     }
     BitSetGroup bitSetGroup = new BitSetGroup(pageNumbers);
-    for (int i = 0; i < pageNumbers; i++) {
-      BitSet set = new BitSet(numberOfRows[i]);
-      RowIntf row = new RowImpl();
-      BitSet prvBitset = null;
-      // if bitset pipe line is enabled then use rowid from previous bitset
-      // otherwise use older flow
-      if (!useBitsetPipeLine ||
-          null == rawBlockletColumnChunks.getBitSetGroup() ||
-          null == bitSetGroup.getBitSet(i) ||
-          rawBlockletColumnChunks.getBitSetGroup().getBitSet(i).isEmpty()) {
-        for (int index = 0; index < numberOfRows[i]; index++) {
-          createRow(rawBlockletColumnChunks, row, i, index);
-          Boolean rslt = false;
-          try {
-            rslt = exp.evaluate(row).getBoolean();
-          }
-          // Any invalid member while evaluation shall be ignored, system will log the
-          // error only once since all rows the evaluation happens so inorder to avoid
-          // too much log inforation only once the log will be printed.
-          catch (FilterIllegalMemberException e) {
-            FilterUtil.logError(e, false);
-          }
-          if (null != rslt && rslt) {
-            set.set(index);
+
+    if (isDimensionPresentInCurrentBlock.length == 1 && isDimensionPresentInCurrentBlock[0]) {
+      // fill default value here
+      DimColumnResolvedFilterInfo dimColumnEvaluatorInfo = dimColEvaluatorInfoList.get(0);
+      // if filter dimension is not present in the current add its default value
+      if (dimColumnEvaluatorInfo.getDimension().getDataType().isComplexType()) {
+        for (int i = 0; i < pageNumbers; i++) {
+          BitSet set = new BitSet(numberOfRows[i]);
+          RowIntf row = new RowImpl();
+          for (int index = 0; index < numberOfRows[i]; index++) {
+            ArrayQueryType complexType =
+                (ArrayQueryType) complexDimensionInfoMap.get(dimensionChunkIndex[i]);
+            int[] numberOfChild = complexType
+                .getNumberOfChild(rawBlockletColumnChunks.getDimensionRawColumnChunks(), null,

Review comment:
       how about to get all numbers of the child once 

##########
File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
##########
@@ -222,49 +224,90 @@ public BitSetGroup applyFilter(RawBlockletColumnChunks rawBlockletColumnChunks,
       }
     }
     BitSetGroup bitSetGroup = new BitSetGroup(pageNumbers);
-    for (int i = 0; i < pageNumbers; i++) {
-      BitSet set = new BitSet(numberOfRows[i]);
-      RowIntf row = new RowImpl();
-      BitSet prvBitset = null;
-      // if bitset pipe line is enabled then use rowid from previous bitset
-      // otherwise use older flow
-      if (!useBitsetPipeLine ||
-          null == rawBlockletColumnChunks.getBitSetGroup() ||
-          null == bitSetGroup.getBitSet(i) ||
-          rawBlockletColumnChunks.getBitSetGroup().getBitSet(i).isEmpty()) {
-        for (int index = 0; index < numberOfRows[i]; index++) {
-          createRow(rawBlockletColumnChunks, row, i, index);
-          Boolean rslt = false;
-          try {
-            rslt = exp.evaluate(row).getBoolean();
-          }
-          // Any invalid member while evaluation shall be ignored, system will log the
-          // error only once since all rows the evaluation happens so inorder to avoid
-          // too much log inforation only once the log will be printed.
-          catch (FilterIllegalMemberException e) {
-            FilterUtil.logError(e, false);
-          }
-          if (null != rslt && rslt) {
-            set.set(index);
+
+    if (isDimensionPresentInCurrentBlock.length == 1 && isDimensionPresentInCurrentBlock[0]) {
+      // fill default value here
+      DimColumnResolvedFilterInfo dimColumnEvaluatorInfo = dimColEvaluatorInfoList.get(0);
+      // if filter dimension is not present in the current add its default value
+      if (dimColumnEvaluatorInfo.getDimension().getDataType().isComplexType()) {
+        for (int i = 0; i < pageNumbers; i++) {
+          BitSet set = new BitSet(numberOfRows[i]);
+          RowIntf row = new RowImpl();
+          for (int index = 0; index < numberOfRows[i]; index++) {
+            ArrayQueryType complexType =

Review comment:
       move to the outside of for loop




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org