You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by eowhadi <gi...@git.apache.org> on 2016/01/08 18:46:15 UTC
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
GitHub user eowhadi opened a pull request:
https://github.com/apache/incubator-trafodion/pull/255
[TRAFODION-1662] Predicate push down revisited (V2)
Currently Trafodion predicate push down to hbase is supporting only the following cases:
<Column><op><Value> AND <Column> <op><value> AND…
And require columns to be “SERIALIZED” (can be compared using binary comparator),
and value data type is not a superset of column data type.
and char type is not case insensitive or upshifted
and no support for Big Numbers
It suffer from several issues:
Handling of nullable column:
When a nullable column is involved in the predicate, because of the way nulls are handled in trafodion (can ether be missing cell, or cell with first byte set to xFF), binary compare cannot do a good job at semantically treating NULL the way a SQL expression would require. So the current behavior is that all null column values as never filtered out and always returned, letting trafodion perform a second pass predicate evaluation to deal with nulls. This can quickly turn counterproductive for very sparse columns, as we would perform useless filtering at region server side (since all nulls are pass), and optimizer has not been coded to turn off the feature on sparse columns.
In addition, since null handling is done on trafodion side, the current code artificially pull all key columns to make sure that a null coded as absent cell is correctly pushed up for evaluation at trafodion layer. This can be optimized by only requiring a single non-nullable column on current code, but this is another story… as you will see bellow, the proposed new way of doing pushdown will handle 100% nulls at hbase layer, therefore requiring adding non nullable column only when a nullable column is needed in the select statement (not in the predicate).
Always returning predicate columns
Select a from t where b>10 would always return the b column to trafodion, even if b is non nullable. This is not necessary and will result in useless network and cpu consumption, even if the predicate is not re-evaluated.
The new advanced predicate push down feature will do the following:
Support any of these primitives:
<col><op><value>
<col><op><col> (nice to have, high cost of custom filter low value after TPC-DS query survey)
Is null
Is not null
Like -> to be investigated, not yet covered in this document
And combination of these primitive with arbitrary number of OR and AND with ( ) associations, given that within () there is only ether any number of OR or any number of AND, no mixing OR and AND inside (). I suspect that normalizer will always convert expression so that this mixing never happen…
And will remove the 2 shortcoming of previous implementation: all null cases will be handled at hbase layer, never requiring re-doing evaluation and the associated pushing up of null columns, and predicate columns will not be pushed up if not needed by the node for other task than the predicate evaluation.
Note that BETWEEN and IN predicate, when normalized as one of the form supported above, will be pushed down too. Nothing in the code will need to be done to support this.
Improvement of explain:
We currently do not show predicate push down information in the scan node. 2 key information is needed:
Is predicate push down used
What columns are retrieved by the scan node (investigate why we get column all instead of accurate information)
The first one is obviously used to determine if all the conditions are met to have push down available, and the second is used to make sure we are not pushing up data from columns we don’t need.
Note that columns info is inconsistently shown today. Need to fix this.
Enablement using an existing ON/OFF CQD (HBASE_FILTER_PREDS) that will be replaced with a multi value CQD that will enable various level of push down optimization, like we have on PCODE optimization level.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/eowhadi/incubator-trafodion predicatePushdownV2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #255
----
commit 1c5f243f7c79e9ceb4a008099e60641d90515037
Author: Eric Owhadi <er...@esgyn.com>
Date: 2016-01-07T01:25:54Z
First commit for advanced predicate pushdown feature (also known as pushdown V2)
associated JIRA TRAFODION-1662 Predicate push down revisited (V2). The JIRA contains a blueprint document, useful to understand what the code is supposed to do.
This code is enabled using CQD hbase_filter_preds '2', and bypassed otherwise. Except for the change implemented in ValueDesc.cpp that is a global bug fix whereValueIdSet are supposed to contain set of valueID ANDed together, and should not contain any ValueID with operator ITM_AND.
commit 8a6f2205c630ff6599eacca247bc4fbe508aa136
Author: Eric Owhadi <er...@esgyn.com>
Date: 2016-01-07T01:34:06Z
Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2
commit 38573bff44e90a4b6bb82d03af8d83631a6e38bb
Author: Eric Owhadi <er...@esgyn.com>
Date: 2016-01-08T01:59:05Z
Merge branch 'master' of github.com:apache/incubator-trafodion into predicatePushdownV2
commit 90795250785f50cc0538284f22b1b8589a84734a
Author: Eric Owhadi <er...@esgyn.com>
Date: 2016-01-08T15:05:38Z
Fix issue where optimization on key column addition should be turned off for MDAM scans, and update EXPECTEDTESTRTS to showcase the new value expected as byte read showing a 53% improvement over previous code
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49481817
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -358,6 +365,417 @@ void resetAutoFlush() {
table.setAutoFlush(true, true);
}
+ private enum Op {
+ EQUAL, EQUAL_NULL, NOT_EQUAL, NOT_EQUAL_NULL, LESS, LESS_NULL, LESS_OR_EQUAL, LESS_OR_EQUAL_NULL, GREATER, GREATER_NULL,
+ GREATER_OR_EQUAL, GREATER_OR_EQUAL_NULL, NO_OP, NO_OP_NULL,IS_NULL, IS_NULL_NULL, IS_NOT_NULL, IS_NOT_NULL_NULL, AND, OR};
+
+ private Filter SingleColumnValueExcludeOrNotFilter(byte[] columnToFilter,
+ CompareOp op,
+ ByteArrayComparable comparator,
+ HashMap<String,Object> columnsToRemove,
+ Boolean... filterIfMissing){
+ Filter result;
+ boolean fMissing = filterIfMissing.length>0?filterIfMissing[0]:false;//default to false
+ if ((columnsToRemove == null) || !columnsToRemove.containsKey(new String(columnToFilter))){
+ result = new SingleColumnValueFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueFilter)result).setFilterIfMissing(fMissing);
+ }
+ else{
+ result= new SingleColumnValueExcludeFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueExcludeFilter)result).setFilterIfMissing(fMissing);
+ }
+ return result;
+ }
+
+ // construct the hbase filter
+ // optimizes for OR and AND associativity
+ // optimizes for detection of a<? and a>? on nullable and non nullable column equivalent to a<>?
+ // optimize for null check factorization (A not null and (A <op> ?)) or (A not null and A <op2> ?) -> A not null and (A <op> ? or A <op2> ?)
+ // this is an important optimzation for IN statement on non null column
+ // uses the columnToRemove parametter to know if we need to use the SingleColumnValue Exclude or not method to limit returned columns
+
+ private Filter constructV2Filter(Object[] colNamesToFilter,
+ Object[] compareOpList,
+ Object[] colValuesToCompare,
+ HashMap<String,Object> columnsToRemove){
+ LinkedList linkedList = new LinkedList();
+ //populate the list with nodes in reverse polish notation order.
+ int k=0;//column index
+ int kk=0;//value index
+ for (int i=1; i<compareOpList.length; i++){ // skip first one containing "V2" marker
+ String opStr = new String((byte[])compareOpList[i]);
+ switch(Op.valueOf(opStr)){
+
+ case EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove
+ ));
+ k++;kk++;
+ break;
+ case EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true //filterIfMissing
+ ),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case NOT_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case NOT_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case LESS:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case LESS_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case LESS_OR_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case LESS_OR_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case GREATER:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case GREATER_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case GREATER_OR_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case GREATER_OR_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case NO_OP:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case NO_OP_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case IS_NULL:
+ // is null on a non nullable column!
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP, //exclude everything
+ new BinaryPrefixComparator((new byte[]{})),
+ columnsToRemove));
+ k++;
+ break;
+ case IS_NULL_NULL:
+ // is_null on nullable column: is absent OR has the first byte set to FF indicating NULL.
+ linkedList.addLast(
+ new FilterList(FilterList.Operator.MUST_PASS_ONE, //OR
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new NullComparator(),//is absent?
+ columnsToRemove),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{-1}),//0xFF has null prefix indicator
+ columnsToRemove)));
+ k++;
+ break;
+ case IS_NOT_NULL:
+ // is not null on a non nullable column!
+ // do nothing, always true
+ k++;
+ break;
+ case IS_NOT_NULL_NULL:
+ // is_not_null on nullable column: is not absent AND has the first byte not set to FF indicating NULL.
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryPrefixComparator(new byte[]{-1}),// 0xFF has null prefix indicator
+ columnsToRemove,
+ true));//filter if missing (if absent null)
+ k++;
+ break;
+ case AND:
+ linkedList.addLast("AND");
+ break;
+ case OR:
+ linkedList.addLast("OR");
+ break;
+ default:
+ }//switch
+ }//for
+ //evaluate the reverse polish notation list
+ while (linkedList.size()>1){// evaluate until only one element is left in the list
+ //look for first operator (AND or OR)
+ int j=0;
+ while (j<linkedList.size() && !(linkedList.get(j) instanceof String)){
+ j++;
+ }
+ //here j points on the first operator; (all operands are of type Filter)
+ if (j==linkedList.size()){logger.error("j==linkedList.size()");return null;} // should not happen
+ Filter leftOperand;
+ Filter rightOperand;
+ switch(Op.valueOf((String)linkedList.get(j))){
+ case AND:
+ FilterList filterListAnd = new FilterList(FilterList.Operator.MUST_PASS_ALL); //AND filterList
+ //left operand
+ leftOperand = (Filter)linkedList.get(j-2);
+ if (leftOperand instanceof FilterList && ((FilterList)leftOperand).getOperator()==FilterList.Operator.MUST_PASS_ALL){//associativity of AND optimization
+ //for(Filter f:((FilterList)leftOperand).getFilters())
+ // filterListAnd.addFilter(f);
+ filterListAnd = (FilterList)leftOperand; //more efficient than the 2 lines above (kept commented out for code lisibility)
+ }else{
+ filterListAnd.addFilter(leftOperand);
+ }
+ // right operand
+ rightOperand = (Filter)linkedList.get(j-1);
+ if (rightOperand instanceof FilterList && ((FilterList)rightOperand).getOperator()==FilterList.Operator.MUST_PASS_ALL){//associativity of AND optimization
+ for(Filter f:((FilterList)rightOperand).getFilters())
+ filterListAnd.addFilter(f);
+ }else{
+ filterListAnd.addFilter(rightOperand);
+ }
+ // setup evaluated filter
+ linkedList.set(j,filterListAnd); // replace the operator with the constructer filter
+ linkedList.remove(j-1);// remove right operand
+ linkedList.remove(j-2);// remove left operand. warning order matter
+ break;
+ case OR:
+ FilterList filterListOr = new FilterList(FilterList.Operator.MUST_PASS_ONE); //OR filterList
+ leftOperand = (Filter)linkedList.get(j-2);
+ rightOperand = (Filter)linkedList.get(j-1);
+ //begin detection of null check factorization (A not null and (A <op> ?)) or (A not null and A <op2> ?) -> A not null and (A <op> ? or A <op2> ?)
+ //the code is doing more than just nullcheck, but any factorization where left operands are identical
+ if (leftOperand instanceof FilterList && rightOperand instanceof FilterList &&
+ ((FilterList)leftOperand).getOperator() == FilterList.Operator.MUST_PASS_ALL &&
+ ((FilterList)rightOperand).getOperator() == FilterList.Operator.MUST_PASS_ALL &&
+ ((FilterList)leftOperand).getFilters().size() == 2 &&
+ ((FilterList)rightOperand).getFilters().size() == 2 &&
+ ((FilterList)leftOperand).getFilters().get(0) instanceof SingleColumnValueFilter && //cannot be SingleColumnValueExcludeFilter when we have the optimization scenario
+ ((FilterList)rightOperand).getFilters().get(0) instanceof SingleColumnValueFilter){//cannot be SingleColumnValueExcludeFilter when we have the optimization scenario
+ SingleColumnValueFilter scvfLeft = (SingleColumnValueFilter)((FilterList)leftOperand).getFilters().get(0);
+ SingleColumnValueFilter scvfRight = (SingleColumnValueFilter)((FilterList)rightOperand).getFilters().get(0);
+ if (scvfLeft.getOperator() == scvfRight.getOperator() && //more general case than just for null check (identical operands)
+ Arrays.equals(scvfLeft.getQualifier(),scvfRight.getQualifier()) &&
+ Arrays.equals(scvfLeft.getFamily(),scvfRight.getFamily()) &&
+ Arrays.equals(scvfLeft.getComparator().getValue(),scvfRight.getComparator().getValue()) &&
+ (scvfLeft.getFilterIfMissing() == scvfRight.getFilterIfMissing())){
+ Filter left = ((FilterList)leftOperand).getFilters().get(1);
+ Filter right = ((FilterList)rightOperand).getFilters().get(1);
+ if (left instanceof FilterList && ((FilterList)left).getOperator()==FilterList.Operator.MUST_PASS_ONE){//associativity of OR optimization
+ //for(Filter f:((FilterList)left).getFilters())
+ // filterListOr.addFilter(f);
+ filterListOr = (FilterList)left; // more efficient than the 2 lines above (kept commented out for code lisibility)
+ }else{
+ filterListOr.addFilter(left);
+ }
+ // right operand
+ if (right instanceof FilterList && ((FilterList)right).getOperator()==FilterList.Operator.MUST_PASS_ONE){//associativity of OR optimization
+ for(Filter f:((FilterList)right).getFilters())
+ filterListOr.addFilter(f);
+ }else{
+ filterListOr.addFilter(right);
+ }
+ linkedList.set(j,new FilterList(FilterList.Operator.MUST_PASS_ALL,scvfLeft,filterListOr));//resulting factorized AND filter
+ linkedList.remove(j-1);// remove right operand
+ linkedList.remove(j-2);// remove left operand. warning order matter
+ break;
+ }
+ }
+ //end detection of null (and more) check factorization
+ //begin detection of RangeSpec a<>? transformed to a<? or a>? to convert it back to a <> ? when we push down
+ //check for <> on non nullable columns
+ if (leftOperand instanceof SingleColumnValueFilter && rightOperand instanceof SingleColumnValueFilter){
+ SingleColumnValueFilter leftscvf = (SingleColumnValueFilter)leftOperand;
+ SingleColumnValueFilter rightscvf = (SingleColumnValueFilter)rightOperand;
+ if (leftscvf.getOperator() == CompareOp.LESS && rightscvf.getOperator()== CompareOp.GREATER &&
+ Arrays.equals(leftscvf.getQualifier(), rightscvf.getQualifier()) &&
+ Arrays.equals(leftscvf.getFamily(), rightscvf.getFamily()) &&
+ Arrays.equals(leftscvf.getComparator().getValue(),rightscvf.getComparator().getValue())
+ ){
+ // setup evaluated filter
+ linkedList.set(j,new SingleColumnValueFilter(leftscvf.getFamily(), leftscvf.getQualifier(), CompareOp.NOT_EQUAL, leftscvf.getComparator())); // replace the operator with the constructer filter
+ linkedList.remove(j-1);// remove right operand
+ linkedList.remove(j-2);// remove left operand. warning order matter
+ break;
+ }
+ }
+ //check for <> on nullable column
+ if( leftOperand instanceof FilterList && rightOperand instanceof FilterList){
+ //no need to check FilterList size, as all possible case FilterList size is at least 2.
+ if (((FilterList)leftOperand).getFilters().get(1) instanceof SingleColumnValueFilter &&
+ ((FilterList)rightOperand).getFilters().get(1) instanceof SingleColumnValueFilter){
+ SingleColumnValueFilter leftscvf = (SingleColumnValueFilter)((FilterList)leftOperand).getFilters().get(1);
+ SingleColumnValueFilter rightscvf = (SingleColumnValueFilter)((FilterList)rightOperand).getFilters().get(1);
+ if (leftscvf.getOperator() == CompareOp.LESS && rightscvf.getOperator()== CompareOp.GREATER &&
+ Arrays.equals(leftscvf.getQualifier(), rightscvf.getQualifier()) &&
+ Arrays.equals(leftscvf.getFamily(), rightscvf.getFamily()) &&
+ Arrays.equals(leftscvf.getComparator().getValue(),rightscvf.getComparator().getValue())
+ ){
+ // setup evaluated filter
+ SingleColumnValueFilter nullCheck = new SingleColumnValueFilter(// null checker
+ leftscvf.getFamily(), leftscvf.getQualifier(),
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}));
+ nullCheck.setFilterIfMissing(true);
+ linkedList.set(j,new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ nullCheck,
+ new SingleColumnValueFilter(
+ leftscvf.getFamily(), leftscvf.getQualifier(),
+ CompareOp.NOT_EQUAL,
+ leftscvf.getComparator())));
+ linkedList.remove(j-1);// remove right operand
+ linkedList.remove(j-2);// remove left operand. warning order matter
+ break;
+ }
+ }
+ }
+ //end detection of RangeSpec a<>?
+ //now general case...
+ //left operand
+ if (leftOperand instanceof FilterList && ((FilterList)leftOperand).getOperator()==FilterList.Operator.MUST_PASS_ONE){//associativity of OR optimization
+ //for(Filter f:((FilterList)leftOperand).getFilters())
+ // filterListOr.addFilter(f);
+ filterListOr = (FilterList)leftOperand; // more efficient than the 2 lines above (kept commented out for code lisibility)
+ }else{
+ filterListOr.addFilter(leftOperand);
+ }
+ // right operand
+ if (rightOperand instanceof FilterList && ((FilterList)rightOperand).getOperator()==FilterList.Operator.MUST_PASS_ONE){//associativity of OR optimization
+ for(Filter f:((FilterList)rightOperand).getFilters())
+ filterListOr.addFilter(f);
+ }else{
+ filterListOr.addFilter(rightOperand);
+ }
+ // setup evaluated filter
+ linkedList.set(j,filterListOr); // replace the operator with the constructer filter
+ linkedList.remove(j-1);// remove right operand
+ linkedList.remove(j-2);// remove left operand. warning order matter
+ break;
--- End diff --
The indentation is not right here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49254591
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
Could we have a buffer overrun here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49256114
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
--- End diff --
Just curious why we throw away the return from extractHbaseFilterPredsV2 and always return 0, when we don't do that in the V1 case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49367232
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
OK will remove all tabs from all files, my bad, did not know... must have been written somewhere by I missed it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49484458
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -358,6 +365,417 @@ void resetAutoFlush() {
table.setAutoFlush(true, true);
}
+ private enum Op {
+ EQUAL, EQUAL_NULL, NOT_EQUAL, NOT_EQUAL_NULL, LESS, LESS_NULL, LESS_OR_EQUAL, LESS_OR_EQUAL_NULL, GREATER, GREATER_NULL,
+ GREATER_OR_EQUAL, GREATER_OR_EQUAL_NULL, NO_OP, NO_OP_NULL,IS_NULL, IS_NULL_NULL, IS_NOT_NULL, IS_NOT_NULL_NULL, AND, OR};
+
+ private Filter SingleColumnValueExcludeOrNotFilter(byte[] columnToFilter,
+ CompareOp op,
+ ByteArrayComparable comparator,
+ HashMap<String,Object> columnsToRemove,
+ Boolean... filterIfMissing){
+ Filter result;
+ boolean fMissing = filterIfMissing.length>0?filterIfMissing[0]:false;//default to false
+ if ((columnsToRemove == null) || !columnsToRemove.containsKey(new String(columnToFilter))){
+ result = new SingleColumnValueFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueFilter)result).setFilterIfMissing(fMissing);
+ }
+ else{
+ result= new SingleColumnValueExcludeFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueExcludeFilter)result).setFilterIfMissing(fMissing);
+ }
+ return result;
+ }
+
+ // construct the hbase filter
+ // optimizes for OR and AND associativity
+ // optimizes for detection of a<? and a>? on nullable and non nullable column equivalent to a<>?
+ // optimize for null check factorization (A not null and (A <op> ?)) or (A not null and A <op2> ?) -> A not null and (A <op> ? or A <op2> ?)
+ // this is an important optimzation for IN statement on non null column
+ // uses the columnToRemove parametter to know if we need to use the SingleColumnValue Exclude or not method to limit returned columns
+
+ private Filter constructV2Filter(Object[] colNamesToFilter,
+ Object[] compareOpList,
+ Object[] colValuesToCompare,
+ HashMap<String,Object> columnsToRemove){
+ LinkedList linkedList = new LinkedList();
+ //populate the list with nodes in reverse polish notation order.
+ int k=0;//column index
+ int kk=0;//value index
+ for (int i=1; i<compareOpList.length; i++){ // skip first one containing "V2" marker
+ String opStr = new String((byte[])compareOpList[i]);
+ switch(Op.valueOf(opStr)){
+
+ case EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove
+ ));
+ k++;kk++;
+ break;
+ case EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true //filterIfMissing
+ ),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case NOT_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case NOT_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case LESS:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case LESS_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case LESS_OR_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case LESS_OR_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case GREATER:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case GREATER_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case GREATER_OR_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case GREATER_OR_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case NO_OP:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case NO_OP_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case IS_NULL:
+ // is null on a non nullable column!
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP, //exclude everything
+ new BinaryPrefixComparator((new byte[]{})),
+ columnsToRemove));
+ k++;
+ break;
+ case IS_NULL_NULL:
+ // is_null on nullable column: is absent OR has the first byte set to FF indicating NULL.
+ linkedList.addLast(
+ new FilterList(FilterList.Operator.MUST_PASS_ONE, //OR
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new NullComparator(),//is absent?
+ columnsToRemove),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{-1}),//0xFF has null prefix indicator
+ columnsToRemove)));
+ k++;
+ break;
+ case IS_NOT_NULL:
+ // is not null on a non nullable column!
+ // do nothing, always true
+ k++;
+ break;
+ case IS_NOT_NULL_NULL:
+ // is_not_null on nullable column: is not absent AND has the first byte not set to FF indicating NULL.
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryPrefixComparator(new byte[]{-1}),// 0xFF has null prefix indicator
+ columnsToRemove,
+ true));//filter if missing (if absent null)
+ k++;
+ break;
+ case AND:
+ linkedList.addLast("AND");
+ break;
+ case OR:
+ linkedList.addLast("OR");
+ break;
+ default:
+ }//switch
+ }//for
--- End diff --
This for loop just puts items in reverse order. Where do we do the in-order traversal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49254349
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
--- End diff --
Subject to buffer overrun. Could use NAString instead to avoid this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49480173
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -358,6 +365,417 @@ void resetAutoFlush() {
table.setAutoFlush(true, true);
}
+ private enum Op {
+ EQUAL, EQUAL_NULL, NOT_EQUAL, NOT_EQUAL_NULL, LESS, LESS_NULL, LESS_OR_EQUAL, LESS_OR_EQUAL_NULL, GREATER, GREATER_NULL,
+ GREATER_OR_EQUAL, GREATER_OR_EQUAL_NULL, NO_OP, NO_OP_NULL,IS_NULL, IS_NULL_NULL, IS_NOT_NULL, IS_NOT_NULL_NULL, AND, OR};
+
+ private Filter SingleColumnValueExcludeOrNotFilter(byte[] columnToFilter,
--- End diff --
comment please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49482410
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
--- End diff --
code compaction vs readability is a long debated topic and end up being developer taste specific. I personally prefer compaction too, but wanted to respect the model I followed to favor consistancy over my personal taste. see the code I refer too:
for (Lng32 i = 0; i < colNameLen; i++)
{
colName[i] = currPtr[currPos];
currPos++;
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49227198
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
Might be a problem with tabs in the code. When I look at this in the apache code review tool, the indentation is all out-of-whack. Just this file though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49609586
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
--- End diff --
OK make sense, will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49488653
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
I personally feel the review process reveals the problems and a rework process follows. The rework mostly has a focus of the code reviewed, and can affect the surrounding code.
Most of the concerns raised are about the new code and should be the focus of rework.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49472503
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
--- End diff --
Need to add a comment to indicate that colVID, valueID and op are output parameters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49477573
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
these are V1 code identical.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49488277
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
--- End diff --
no, matter of taste. I prefer it too, but compiler will optimize the same. Some developer prefer the readability of separating, and some will even go as far as doing currPos = currPos + 1 because they dislike the ++ syntax. The snippet above is not my code, it is the code I modeled mine on, to maximize code consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49481705
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
as stated above this is a function crated out of the model in ComTdbHbaseAccess.cpp. So any refactoring / comment suggestion would be on the JIRA to refactor/fix the buffer overrun in a consistant manner
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49478179
--- Diff: core/sql/optimizer/ValueDesc.cpp ---
@@ -3165,7 +3165,11 @@ void ValueIdSet::replaceVEGExpressions
if (iePtr != exprId.getItemExpr()) // a replacement was done
{
subtractElement(exprId); // remove existing ValueId
- newExpr += iePtr->getValueId(); // replace with a new one
+ //insert new expression(s)
+ if (iePtr->getOperatorType() == ITM_AND)
+ iePtr->convertToValueIdSet(newExpr, NULL, ITM_AND, FALSE, FALSE);
--- End diff --
Should add a comment to ValueIdSet::replaceVEGExpressions() on converting the AND backbone into a valueId set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49462687
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
Some comments please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/255
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49504887
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
--- End diff --
On the java side, without changing the function signature and all the boiler plate that comes with JNI?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49474337
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
--- End diff --
the hasColReference is use on the opposite side of the expression basically to filter out anything like <col><op>function(<col>), because we only support <col><op><val>. So this is correct and this is V1 code too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49411448
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
--- End diff --
This also needs to check for !static_cast<BiRelat *>(ie)->getSpecialNulls(). If the "special nulls" flag is set, then the comparison logic is different and includes comparing the NULL indicator. The HBase filters added here just check for non-null values. Special nulls are usually used when comparing index columns and base columns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49638631
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
I'm puzzled by this code. It strips trailing \0 characters from the binary value of a constant. \0 is valid in almost any constant I can think of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49487987
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
Should we strive to write better code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49470458
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
--- End diff --
correct that is what I said when we talked about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49621888
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
an additional concern I have is that given the code is big, splitting in 2 will make it twice as big from a code binary size standpoint, affecting L1 code cache efficiency. There is no way optimizer would compile it back to the original form.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49478515
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
--- End diff --
to leverage the same structure and make the code and changes a lot simpler. I have seen re-use of pointers with different names in the code to serve the same purpose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49508239
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
:-). Reviewing code instead of inspecting does bring out a sense of urgency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49412332
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
No need to check for ITM_REFERENCE, those get eliminated in the binder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49604527
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
Hi,
Morning.
I have worked on the separation of the original extractHbaseFilterPredsV2()
into two routines: isFeasibleToExtractHbaseFilterPredsV2()
and extractHbaseFilterPredsV2(). Please review the attached two files.
While this realizes the objective of one function does one thing only, I
still feel both new functions can be simplified further, in that there is
still striking level of similarities between the AND and OR branches.
Sounds like the two branches can be reduced into 1.
Thanks --Qifan
On Tue, Jan 12, 2016 at 4:41 PM, Hans Zeller <no...@github.com>
wrote:
> In core/sql/src/main/java/org/trafodion/sql/HTableClient.java
> <https://github.com/apache/incubator-trafodion/pull/255#discussion_r49526031>
> :
>
> > + FilterList resultColumnsOnlyFilter = null;
> > + if (narrowDownResultColumns){
> > + HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
> > + // initialize hm with list of columns requested for output
> > + for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
> > + if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
> > + hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
> > + }else{
> > + ArrayList<byte[]> al = new ArrayList<byte[]>();
> > + al.add((byte[])columns[i]);
> > + hm.put(new String(getFamily((byte[])columns[i])), al);
> > + }
> > + }
> > +
> > + if (hm.size()==1){//only one column family
> > + resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
>
> Thank you for the explanation!
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-trafodion/pull/255/files#r49526031>.
>
--
Regards, --Qifan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49476708
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
+ ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
+
+ workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
+ .setDataPointer(hbaseFilterValRow_);
+
+ ex_expr::exp_return_type evalRetCode =
+ hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
+ if (evalRetCode == ex_expr::EXPR_ERROR)
+ {
+ return -1;
+ }
+
+ ExpTupleDesc * hfrTD =
+ hbaseAccessTdb().workCriDesc_->getTupleDescriptor
+ (hbaseAccessTdb().hbaseFilterValTuppIndex_);
+
+ hbaseFilterValues_.clear();
+ for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
--- End diff --
this is existing code, comment is already added on the line of code added (2811). 2855 is the closing of it...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49370819
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
--- End diff --
code structure is consistent with predicate push down V1 structure. Was copy pasted and modified. So would not want to introduce code inconsistency
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49476810
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11337,6 +11816,38 @@ RelExpr * HbaseAccess::preCodeGen(Generator * generator,
if (! FileScan::preCodeGen(generator,externalInputs,pulledNewInputs))
return NULL;
+ //compute isUnique:
+ NABoolean isUnique = FALSE;
+ if (listOfRangeRows_.entries() == 0)
+ {
+ if ((searchKey() && searchKey()->isUnique()) &&
+ (listOfUniqueRows_.entries() == 0))
+ isUnique = TRUE;
+ else if ((NOT (searchKey() && searchKey()->isUnique())) &&
+ (listOfUniqueRows_.entries() == 1) &&
+ (listOfUniqueRows_[0].rowIds_.entries() == 1))
+ isUnique = TRUE;
+ }
+
+ // executorPred() contains an ANDed list of predicates.
+ // if hbase filter preds are enabled, then extracts those preds from executorPred()
+ // which could be pushed down to hbase.
+ // Do this only for non-unique scan access.
+ ValueIdSet newExePreds;
+ ValueIdSet* originExePreds = new (generator->wHeap())ValueIdSet(executorPred()) ;//saved for futur nullable column check
+
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) != DF_MINIMUM){ // the check for V2 and above is moved up before calculating retrieved columns
+ if ((NOT isUnique) &&
--- End diff --
The computation on isUnique can be avoided if it is not for V2 push-down. Suggest to move the CQD check earlier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49407054
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
Might be good to open a JIRA and leave the code as-is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49410393
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
--- End diff --
We talked about this, a ValueIdSet should never contain an AND, so I would suggest deleting the code for ITM_AND.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49511754
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
+ opList_.insert("AND"); // insert an AND node since both side are OK to push down
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we just pushed it down, so increase the node count pushed down.
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (leftOK){ // if only left is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate left tree
+ newExePreds.insert(rightPreds); //make sure we add the right child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we pushed down left side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (rightOK){// if only right is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate right tree
+ newExePreds.insert(leftPreds);//make sure we add the left child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++;// we pushed down right side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else{
+ if(!checkOnly){
+ newExePreds.insert(vid);// we pushed down nothing, make sure the whole node is evaluated by Executor
+ }
+ if (preds.entries()==1)
+ return FALSE;
+
+ }
+ }
+ else{//if not first AND layer, both left and right must be pushable to get anything pushed
+ if(extractHbaseFilterPredsV2(generator, leftPreds, newExePreds, TRUE, FALSE)&&
--- End diff --
Well, we had a lot of problems with that. The compiler does have methods to convert ORs into a join, but they are not always used, since they don't always give the best performance. There are easy methods to do the iteration. Class ItemExprList (file optimizer/ItemExprList.h) can do that. Construct such a object, passing ie and ITM_AND or ITM_OR as the backbone type and it will make a list of all the conjuncts or disjuncts. I have to admit that this object will also do a recursive call, but with a lot less local variables on the stack.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49473289
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
--- End diff --
Not sure I understand. This should be a problem in V1 too if it is a problem. But you are saying this case is for comparing index column and base columns, so this would be out of scope, since we are only supporting <column><op><value> and not <column><op><column> ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49373886
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
--- End diff --
That's also V1 code. Nothing special with V2. copy pasted...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49481413
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -358,6 +365,417 @@ void resetAutoFlush() {
table.setAutoFlush(true, true);
}
+ private enum Op {
+ EQUAL, EQUAL_NULL, NOT_EQUAL, NOT_EQUAL_NULL, LESS, LESS_NULL, LESS_OR_EQUAL, LESS_OR_EQUAL_NULL, GREATER, GREATER_NULL,
+ GREATER_OR_EQUAL, GREATER_OR_EQUAL_NULL, NO_OP, NO_OP_NULL,IS_NULL, IS_NULL_NULL, IS_NOT_NULL, IS_NOT_NULL_NULL, AND, OR};
+
+ private Filter SingleColumnValueExcludeOrNotFilter(byte[] columnToFilter,
+ CompareOp op,
+ ByteArrayComparable comparator,
+ HashMap<String,Object> columnsToRemove,
+ Boolean... filterIfMissing){
+ Filter result;
+ boolean fMissing = filterIfMissing.length>0?filterIfMissing[0]:false;//default to false
+ if ((columnsToRemove == null) || !columnsToRemove.containsKey(new String(columnToFilter))){
+ result = new SingleColumnValueFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueFilter)result).setFilterIfMissing(fMissing);
+ }
+ else{
+ result= new SingleColumnValueExcludeFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueExcludeFilter)result).setFilterIfMissing(fMissing);
+ }
+ return result;
+ }
+
+ // construct the hbase filter
+ // optimizes for OR and AND associativity
+ // optimizes for detection of a<? and a>? on nullable and non nullable column equivalent to a<>?
+ // optimize for null check factorization (A not null and (A <op> ?)) or (A not null and A <op2> ?) -> A not null and (A <op> ? or A <op2> ?)
+ // this is an important optimzation for IN statement on non null column
+ // uses the columnToRemove parametter to know if we need to use the SingleColumnValue Exclude or not method to limit returned columns
+
+ private Filter constructV2Filter(Object[] colNamesToFilter,
+ Object[] compareOpList,
+ Object[] colValuesToCompare,
+ HashMap<String,Object> columnsToRemove){
+ LinkedList linkedList = new LinkedList();
+ //populate the list with nodes in reverse polish notation order.
+ int k=0;//column index
+ int kk=0;//value index
+ for (int i=1; i<compareOpList.length; i++){ // skip first one containing "V2" marker
+ String opStr = new String((byte[])compareOpList[i]);
+ switch(Op.valueOf(opStr)){
+
+ case EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove
+ ));
+ k++;kk++;
+ break;
+ case EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true //filterIfMissing
+ ),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
--- End diff --
Another help function to return the NOT semantics can be useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49471229
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
--- End diff --
need to check if the result of the bindNode()/preCodeGen() is a NULL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49227922
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
New code only; the old code indentation looks fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49507863
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -754,16 +1017,30 @@ HbaseAccess::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
}
+ // get column retrieved
+ if (((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames()){
+ description += "column_retrieved: ";
+ appendListOfColumns(((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames(),tdb,description);
+ }
+ // get predicate pushed down in Reverse Polish Notation for the AND / OR operators.
+ // could transform it standard notation for better readability, but good enough for now...
+ // could also evaluate the constants instead of hard coded ?, but good enough for now...
+ if (((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames()){
+ description += "pushed_down_rpn: ";
--- End diff --
Yes, I think to provide accurate description in the plan is important.
In the ORC push down, we use the term orc_search_arguments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49485143
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
+ opList_.insert("AND"); // insert an AND node since both side are OK to push down
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we just pushed it down, so increase the node count pushed down.
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (leftOK){ // if only left is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate left tree
+ newExePreds.insert(rightPreds); //make sure we add the right child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we pushed down left side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (rightOK){// if only right is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate right tree
+ newExePreds.insert(leftPreds);//make sure we add the left child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++;// we pushed down right side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else{
+ if(!checkOnly){
+ newExePreds.insert(vid);// we pushed down nothing, make sure the whole node is evaluated by Executor
+ }
+ if (preds.entries()==1)
+ return FALSE;
+
+ }
+ }
+ else{//if not first AND layer, both left and right must be pushable to get anything pushed
+ if(extractHbaseFilterPredsV2(generator, leftPreds, newExePreds, TRUE, FALSE)&&
--- End diff --
I had the same fear but was thinking that compiler must have a reasonable limit to not push too big or list and turn it into a hash check instead of cycling to many values by for example turning things into hash join? My assumption is not correct? If not the solution will not be to turn that function into no recursive, but block the push down all together, as it is going to be counter productive. Having a huge list of interpreted OR filter is definitely not a good idea and should not be pushed down. Instead, we would want to do a implement a new hbase filter to do hash style filter, or dicotomic check, all in one single filter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49620434
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
Sorry, I can't find the files Qifan mentioned. What Eric mentions is my concern as well, if the two methods have similar structure and need to be synchronized, then wouldn't it be better to keep them as one method? Again, I'm saying that without seeing the new code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49473542
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
--- End diff --
Just wonder why we need insert V2 into the opList_. Since two different functionalities (say v1 and v2) can always be carried out through different function calls all the way to Java and HBase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49607651
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
+ {
+ colFam[jj] = currPtr2[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr2[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ str_sprintf(buf2, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ }else{//native hbase table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ char * colNamePtr1 = NULL;
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colName[500];
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr2[currPos];
+ currPos++;
+ }
+ colName[colNameLen] = 0;
+ colNamePtr1 = colName;
+ str_sprintf(buf2, "%s",colNamePtr1);
+ }
+ pushedDownColumns->advance();
+ }
+
+
+ char* colNamePtr = buf2;
+ if(strcmp(currPtr,"EQUAL")==0){
+ str_sprintf(buf, "(%s=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NOT_EQUAL")==0){
+ str_sprintf(buf, "(%s!=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"LESS")==0){
+ str_sprintf(buf, "(%s<?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if(strcmp(currPtr,"LESS_OR_EQUAL")==0){
+ str_sprintf(buf, "(%s<=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER")==0){
+ str_sprintf(buf, "(%s>?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER_OR_EQUAL")==0){
+ str_sprintf(buf, "(%s>=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NO_OP")==0){//should never happen
+ str_sprintf(buf, "(%s??)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NOT_EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s!=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"LESS_NULL")==0){
+ str_sprintf(buf, "(%s<.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"LESS_OR_EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s<=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER_NULL")==0){
+ str_sprintf(buf, "(%s>.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER_OR_EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s>=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NO_OP_NULL")==0){
+ str_sprintf(buf, "(%s?.?)",colNamePtr);//should never happen
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NULL")==0){
+ str_sprintf(buf, "(%s is_null)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NULL_NULL")==0){
+ str_sprintf(buf, "(%s is_null.)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NOT_NULL")==0){
+ str_sprintf(buf, "(%s is_not_null)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NOT_NULL_NULL")==0){
+ str_sprintf(buf, "(%s is_not_null.)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"AND")==0)
+ outNAString += "AND";
+ else if (strcmp(currPtr,"OR")==0)
+ outNAString += "OR";
+
--- End diff --
that would be a bug in the code. There should never be any not all above. So this bug will translate into not displaying any new operator that we failed to add in the display code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49470340
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
With V2, you will see that the code that is used to compute columns is now happening after the code that is computing the predicate. Therefore the column list only contains the column needed in the returned result set. Where in V1 or no pushdown, the columns contain both the columns needed by the returned result set and the one needed for predicate evaluation. There is no need to have a second list to store the one needed for predicate evaluation, as this information is inherently available in the colNameToFilter. That is why you see that with V2, I add the columns with the colNameToFilter to pass to the scan object, while with V1, colname is passed as is. For the resulting filter, well I am not sure if n**2 means n*2, where n is the columns in the result set only. But yes that is the price to pay to not get the whole thing, and I was indeed concerned about this, that is why I also implemented the SingleColumnValueExcludeFilter alternative whenever possible (this one is not addi
ng any extra filter to do the job). Knowing when the filter is counter productive vs not could be part of a JIRA with the theme of how compiler can decide when push down is good vs counter productive. I intuit that given the poor efficiency of interpreted filter on the java side, vs high efficiency of native code on ESP side, for low selectivity predicate, pushdown will be counter productive... except may be for the column optimization, that should only significantly shine when huge amount of data have to make it to ESP... Anyway, given pred pushdown is disabled by default for now, it is imho OK to release with this, and condition enablement when we have compiler smartness to know when to turn it on or off or partially on (meaning don't try to optimize column)...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49489033
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
--- End diff --
consistent with the code I modeled mine on. And here my personal feeling agrees with the model, as readability is more easy when you read currPtr2[currPos] != ':' than whatever function name we can choose to describe imho. Given the simplicity of the function, compiler will anyway "inline" to avoid function call overhead, so runtime efficiency is not a deciding factor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49255940
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
--- End diff --
I'm guessing that HBase doesn't use null-termination? It might make sense to take one trailing '\0', but is it possible we are sending down a binary string with nulls in it? I'm wondering if this code will handle that correctly by removing all trailing '\0's? (But maybe HBase has null-padding semantics? If it does then this should not matter.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49503316
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
--- End diff --
I can understand your point. Dave, Hans do you also believe that splitting the big function in 2 , one with check on, one with check off would be better?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49467156
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
--- End diff --
In ORC push-down, we have a function to detect whether a valueId is associated with an expression referencing a column. The code can be shared here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49468391
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
--- End diff --
Same comment on code duplication.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49471077
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
this can be better coded with the NAString::strip().
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49513243
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11412,9 +11924,54 @@ RelExpr * HbaseAccess::preCodeGen(Generator * generator,
}
}
- // add all the key columns. If values are missing in hbase, then atleast the key
+ // add key columns. If values are missing in hbase, then atleast the key
// value is needed to retrieve a row.
- HbaseAccess::addColReferenceFromVIDlist(getIndexDesc()->getIndexKey(), retColRefSet_);
+ //only if needed. If there is already a non nullable non added non nullable with default columns in the set, we should not need to add
+ //any other columns.
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MEDIUM && getMdamKeyPtr() == NULL){ //only enable column retrieval optimization with DF_MEDIUM and not for MDAM scan
+ bool needAddingNonNullableColumn = true; //assume we need to add one non nullable column
+ for (ValueId vid = retColRefSet_.init();// look for each column in th eresult set if one match the criteria non null non added non nullable with default
+ retColRefSet_.next(vid);
+ retColRefSet_.advance(vid))
+ {
+ if (originExePreds->isNotNullable(vid)){// it is non nullable
+ NAColumn * nac;
+ switch (vid.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)vid.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)vid.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && !(nac->isAddedColumn() && nac->getDefaultValue())){//check if added and with default... notgood
+ needAddingNonNullableColumn = false; // we found one column meeting all criteria
--- End diff --
Actually, this could use `vid.getNAColumn(TRUE)` to eliminate part of the above code to compute nac.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49465774
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -754,16 +1017,30 @@ HbaseAccess::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
}
+ // get column retrieved
+ if (((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames()){
+ description += "column_retrieved: ";
+ appendListOfColumns(((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames(),tdb,description);
+ }
+ // get predicate pushed down in Reverse Polish Notation for the AND / OR operators.
+ // could transform it standard notation for better readability, but good enough for now...
+ // could also evaluate the constants instead of hard coded ?, but good enough for now...
+ if (((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames()){
+ description += "pushed_down_rpn: ";
--- End diff --
"pusheddown_preds_in_RPN" probably is more accurate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49367695
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
Is the resultColumnsOnlyFilter really necessary, since we already added the columns to the Scan on line 834?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49463189
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
--- End diff --
yes. use of snprintf() will prevent overrun. We once did this as an exercise to change all instances of non-snprintf() to snprintf(). It is a requirement now to use snprintf().
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49478801
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
--- End diff --
adding a V2 makes the code easier to read
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49487905
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
--- End diff --
but that code is wrong as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49226137
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
can you elaborate and compare with the original file please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49254620
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
+ {
+ colFam[jj] = currPtr2[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr2[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ str_sprintf(buf2, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ }else{//native hbase table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ char * colNamePtr1 = NULL;
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colName[500];
--- End diff --
Can this buffer be overrun?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49488554
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
That's great. Just to be sure we talk about the same thing: select a, b from t where b > 10 and c < 5; In this example, the scan uses columns a, b, c and the qualifier filter selects out only a and b? How do we know that the parent of the scan operator requires only a and b? When I wrote n**2 I should have said 1/2 n**2 (assuming non-aligned format). Let's say we select all n columns, meaning we generate n QualifierFilters. Each row has n cells and on each cell we have to evaluate approximately n/2 QualifierFilters until we find one that evaluates to TRUE. That's the worst case, I know, if we select fewer columns or missing columns it will be less.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49470702
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
--- End diff --
let me think about it, I think you are may be right...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49487630
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
--- End diff --
hasColReference() does not filter out functions. I think it would return FALSE for expressions like sin(0.5) or ? + 3 > 10. If we generate an expression to compute this value before passing it to the filter then that's ok. I didn't see that code generation in your change, probably it is already done in the existing V1 logic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49479048
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
--- End diff --
this function is way too big to duplicate. it will be error prone to split in 2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49818100
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
Hans, I think you did not see the code is only active if addedForLikePred, not for all constant. This make sense to avoid the meaningless ending 0 does not interfere with binary compare semantic that will happen on the hbase side when comparing strings associated with an exploded Like predicate. Is that making sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49254562
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
--- End diff --
We'll never have a column name of "V2" at this level will we? (That is, V2 is always unambiguous here?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49505693
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
Why not just fix it by using snprintf()? It would take much less time to do that than to discuss it or file the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49391405
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
The resultColumnsOnlyFilter is use to filter down columns that we don't need in the result set. On line 834, we have all the column that the scan operator will fetch from RS, but all the one that are only needed for predicate evaluation only can be filtered out and not returned to ESP. So resultColumnsOnlyFilter is one way to do this filtering down, along with the other method using SingleColumnValueExcludeFilter instead of the non exclude one. Here is a copy past of the section in blueprint describing the trick:
Always returning predicate columns:
The current pushdown feature only make use of SingleColumnValueFilter. However, there is a sibling filter, SingleColumnValueExcludeFilter, that will behave the same, except that it will not return the column on which it is doing the filtering. Correctly choosing the Exclude version or the regular version is the key to correctly handling the pushing or not of the column participating in the filter. This trick alone is not sufficient, as during complex expression evaluations, some nodes may not be evaluated due to “fast exit”. Therefore, combining results using a FilterList using ArrayList to preserve order and MUST_PASS_ALL using the original Filter expression as first filter, and a FilterList MUST_PASS_ONE of QualifierFilter that will filter in every columns that are needed in the result set. For multi column family table, need to add FamilyFilter in the logic.
To simplify the logic when deciding if we will use the “Exclude” method, or the FamilyFilter/QualifierFilter, we are simply going to use the optimized Exclude method if and only if all of the column added in the result set show up only once in the predicate push down. If at least one shows up twice, we will fall back on Family/QualifierFilter. Also we will check that the predicate does not contain any IS_NULL_NULL operation, as these are transformed into OR filter list, making the Exclude method inappropriate, as we are not guaranteed that all filter inside the OR will be executed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49516999
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
--- End diff --
Let me look into it tomorrow morning to see how to make a separation. The
complexity of the function is a concern to me.
On Tue, Jan 12, 2016 at 3:12 PM, Hans Zeller <no...@github.com>
wrote:
> In core/sql/generator/GenPreCode.cpp
> <https://github.com/apache/incubator-trafodion/pull/255#discussion_r49514116>
> :
>
> > + ItemExpr * ie = vid.getItemExpr();
> > +
> > + // if it is AND operation, recurse through left and right children
> > + if (ie->getOperatorType() == ITM_AND)
> > + {
> > + ValueIdSet leftPreds;
> > + ValueIdSet rightPreds;
> > + leftPreds += ie->child(0)->castToItemExpr()->getValueId();
> > + rightPreds += ie->child(1)->castToItemExpr()->getValueId();
> > + if (isFirstAndLayer){
> > + NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
> > + NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
> > + if (leftOK && rightOK){
> > + if (!checkOnly){
> > + extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
> > + extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
>
> I can't really see how it could be split into a checking and an executing
> method and how that would help. Unless I see a nice alternative I think I'm
> fine with this single method, and deleting the top-level AND code will also
> make the method less complex.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-trafodion/pull/255/files#r49514116>.
>
--
Regards, --Qifan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49370248
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
--- End diff --
nice catch, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49367740
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
I got that limit and code snippet from the file ComTdbHbaseAccess.cpp: static void showStrColNames(Queue * listOfColNames, Space * space,
NABoolean nullTerminated = FALSE)
{
char buf[1000];
listOfColNames->position();
for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
{
char * currPtr = (char*)listOfColNames->getCurr();
char * colNamePtr = NULL;
if (nullTerminated)
{
colNamePtr = currPtr;
}
else
{
Lng32 currPos = 0;
short colNameLen = *(short*)currPtr;
currPos += sizeof(short);
char colName[500];
for (Lng32 i = 0; i < colNameLen; i++)
{
colName[i] = currPtr[currPos];
currPos++;
}
colName[colNameLen] = 0;
colNamePtr = colName;
}
str_sprintf(buf, " Entry #%d: %s",
j+1,
colNamePtr);
space->allocateAndCopyToAlignedSpace(buf, str_len(buf), sizeof(short));
listOfColNames->advance();
} // for
}
So 3 options: 1/ I fix it on my code and we have code inconsistency. 2/ I leave is as is and open a JIRA so that is get fixed in both place: That way we have no inconsistency 3/ someone knows that original code was fine and limit is reasonable? and we don t touch any of it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49479160
--- Diff: core/sql/regress/executor/FILTER140 ---
@@ -0,0 +1,42 @@
+#! /bin/sh
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+
+# Like the pattern-masking of QACOMP, here we filter out such stuff as
+# transaction id. Called by the runregr script before doing diff.
+
+fil=$1
+sed "
+s/^\([ ]*affinity_value[ ]*[\.]*\) .*/affinity_value removed/g
+s/^\([ ]*max_max_cardinality[ ]*[\.]*\) .*/max_max_cardinality removed/g
+s/^\([ ]*PLAN_ID[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*HBASE_PORT[ ]*[\.]*\) .*/ HBASE_PORT........ removed/g
+s/^\([ ]*HBASE_SERVER[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*HBASE_INTERFACE[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*HBASE_SERIALIZATION[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*HBASE_FILTER_PREDS[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*TRAF_ALIGNED_ROW_FORMAT[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*TRAF_INDEX_CREATE_OPT[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*embedded_arkcmp[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*EST_TOTAL_COST[ ]*[\.]*\) .*/\1 removed/g
+s/^\([ ]*EST_OPER_COST[ ]*[\.]*\) .*/\1 removed/g
+s/VOLATILE_SCHEMA_MXID[0-9]*/VOLATILE_SCHEMA_00/
+" $fil
--- End diff --
Did this file get truncated accidentally? It has a strange ending.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49505122
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
--- End diff --
Yes, I also usually like the separation better, although I use them both, probably depends on how I feel on a particular day...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49485101
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
--- End diff --
Which is the reason for a split. Sorry.
On Tue, Jan 12, 2016 at 10:44 AM, Eric Owhadi <no...@github.com>
wrote:
> In core/sql/generator/GenPreCode.cpp
> <https://github.com/apache/incubator-trafodion/pull/255#discussion_r49479048>
> :
>
> > + ItemExpr * ie = vid.getItemExpr();
> > +
> > + // if it is AND operation, recurse through left and right children
> > + if (ie->getOperatorType() == ITM_AND)
> > + {
> > + ValueIdSet leftPreds;
> > + ValueIdSet rightPreds;
> > + leftPreds += ie->child(0)->castToItemExpr()->getValueId();
> > + rightPreds += ie->child(1)->castToItemExpr()->getValueId();
> > + if (isFirstAndLayer){
> > + NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
> > + NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
> > + if (leftOK && rightOK){
> > + if (!checkOnly){
> > + extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
> > + extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
>
> this function is way too big to duplicate. it will be error prone to split
> in 2.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-trafodion/pull/255/files#r49479048>.
>
--
Regards, --Qifan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49627548
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
--- End diff --
checked with Anoop, all of these are not serialized so should not be pushed down. The code already takes care of this check in both V1 and V2 with the invoke of columnEnabledForSerialization(colVID.getItemExpr())))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49621365
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
Hi,
I forwarded the new code in a separate email.
Synchronization probably is not a concern since the two functions do things
differently.
It is also very possible to merge the AND and OR two branches into one.
Thanks --Qifan
On Wed, Jan 13, 2016 at 11:21 AM, Hans Zeller <no...@github.com>
wrote:
> In core/sql/src/main/java/org/trafodion/sql/HTableClient.java
> <https://github.com/apache/incubator-trafodion/pull/255#discussion_r49620434>
> :
>
> > + FilterList resultColumnsOnlyFilter = null;
> > + if (narrowDownResultColumns){
> > + HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
> > + // initialize hm with list of columns requested for output
> > + for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
> > + if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
> > + hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
> > + }else{
> > + ArrayList<byte[]> al = new ArrayList<byte[]>();
> > + al.add((byte[])columns[i]);
> > + hm.put(new String(getFamily((byte[])columns[i])), al);
> > + }
> > + }
> > +
> > + if (hm.size()==1){//only one column family
> > + resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
>
> Sorry, I can't find the files Qifan mentioned. What Eric mentions is my
> concern as well, if the two methods have similar structure and need to be
> synchronized, then wouldn't it be better to keep them as one method? Again,
> I'm saying that without seeing the new code.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-trafodion/pull/255/files#r49620434>.
>
--
Regards, --Qifan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49360592
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
Yes, this is a problem with tab settings. When using your favorite editor, it's very important to set it to use 8 spaces for a tab, otherwise it will mess up the indentation for everyone else (and the indentation of many existing files will be wrong). The editor should also be set to insert spaces, not tabs, into the source file. Please do not check in any files with the wrong tab setting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49526031
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
Thank you for the explanation!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49490258
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
--- End diff --
so you confirm this can happen in <col><op><val> scenario? not only <col><op><col>? if so I'll put it in both place, thanks for catching that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49255334
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
--- End diff --
This code is fine, but I wonder if it could be streamlined/shortened/simplified, by combining some of the if's? For example, if (child0 is ITM_BASECOLUMN or ITM_INDEXCOLUMN or ITM_REFERENCE etc.) && (NOT hasColReference(ie->child(1))))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49624997
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
Looked at the code and that confirms what I said in the previous post.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49508115
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
I feel the refactoring should also be considered in the rework.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49407306
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
I made this comment because I don't see the code reducing the "columns" list. Instead, as far as I can tell, it will add one QualifierFilter for every column in "columns" - maybe I missed something. For a table with n columns, that is n**2 filter evaluations for every row, could get expensive. Also, do we have enough information to tell which columns are required by the scan (after applying filters)? I would have thought that requires a second list of columns as input.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49469476
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
--- End diff --
The above code to check if an added column has default value attribute should be moved to NAColumn as a method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49505468
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
--- End diff --
Yes, I don't have a test case right now, but this condition is easy to check and the design of the "special nulls" feature is definitely not restricted to col op col. You will find other checks for this flag elsewhere in the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49486245
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
--- End diff --
With the "special nulls" semantics, a NULL value is considered equal to another NULL value, and the filters generated don't have that semantic, therefore we need to exclude operators with the "special nulls" semantics. That needs to be done in the V1 code as well if it is missing there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49512451
--- Diff: core/sql/optimizer/ValueDesc.cpp ---
@@ -3165,7 +3165,11 @@ void ValueIdSet::replaceVEGExpressions
if (iePtr != exprId.getItemExpr()) // a replacement was done
{
subtractElement(exprId); // remove existing ValueId
- newExpr += iePtr->getValueId(); // replace with a new one
+ //insert new expression(s)
+ if (iePtr->getOperatorType() == ITM_AND)
+ iePtr->convertToValueIdSet(newExpr, NULL, ITM_AND, FALSE, FALSE);
--- End diff --
My bad, I provided that fix. What about the following, above line 3170: The replacement of a RangeSpec could be an AND, convert ANDed predicates into additional values in newExpr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49369885
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
+ {
+ colFam[jj] = currPtr2[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr2[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ str_sprintf(buf2, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ }else{//native hbase table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ char * colNamePtr1 = NULL;
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colName[500];
--- End diff --
same as previews, will do what we decide in the 3 options
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49465282
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
+ {
+ colFam[jj] = currPtr2[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr2[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ str_sprintf(buf2, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ }else{//native hbase table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ char * colNamePtr1 = NULL;
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colName[500];
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr2[currPos];
+ currPos++;
+ }
+ colName[colNameLen] = 0;
+ colNamePtr1 = colName;
+ str_sprintf(buf2, "%s",colNamePtr1);
+ }
+ pushedDownColumns->advance();
+ }
+
+
+ char* colNamePtr = buf2;
+ if(strcmp(currPtr,"EQUAL")==0){
+ str_sprintf(buf, "(%s=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NOT_EQUAL")==0){
+ str_sprintf(buf, "(%s!=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"LESS")==0){
+ str_sprintf(buf, "(%s<?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if(strcmp(currPtr,"LESS_OR_EQUAL")==0){
+ str_sprintf(buf, "(%s<=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER")==0){
+ str_sprintf(buf, "(%s>?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER_OR_EQUAL")==0){
+ str_sprintf(buf, "(%s>=?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NO_OP")==0){//should never happen
+ str_sprintf(buf, "(%s??)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NOT_EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s!=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"LESS_NULL")==0){
+ str_sprintf(buf, "(%s<.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"LESS_OR_EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s<=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER_NULL")==0){
+ str_sprintf(buf, "(%s>.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"GREATER_OR_EQUAL_NULL")==0){
+ str_sprintf(buf, "(%s>=.?)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"NO_OP_NULL")==0){
+ str_sprintf(buf, "(%s?.?)",colNamePtr);//should never happen
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NULL")==0){
+ str_sprintf(buf, "(%s is_null)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NULL_NULL")==0){
+ str_sprintf(buf, "(%s is_null.)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NOT_NULL")==0){
+ str_sprintf(buf, "(%s is_not_null)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"IS_NOT_NULL_NULL")==0){
+ str_sprintf(buf, "(%s is_not_null.)",colNamePtr);
+ outNAString += buf;
+ }
+ else if (strcmp(currPtr,"AND")==0)
+ outNAString += "AND";
+ else if (strcmp(currPtr,"OR")==0)
+ outNAString += "OR";
+
--- End diff --
do we need to handle the "not all above" case here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49607906
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
--- End diff --
OK good catch I will add
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49491802
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
OK that make sense, given V1 should go away, consistency with V1 should not matter. For the other consistency (not with V1 but other code, like the buffer ovverun, are you OK with what we decided with Dave (file a separe JIRA and fix both consistantly, while keep bad practice in this checkin, to avoid having a massive PR that contains code clean up that has nothing to do with the feature?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49385099
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
--- End diff --
because in V1,extractHBaseFilterPreds is using the return value to show if there was an error in the function (actually the code never have a path where it would return an error, but by the construct I am sure this is what was intented. in V2, extractHBaseFilterPredsV2 return is used in the recursive call to carry significant code path change. so I force success (return 0) outside on the V2 case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49221353
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
The indentation in this file seems inconsistent, making the logic harder to understand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49369265
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
--- End diff --
Same question for AND and OR, but we are OK as the column nmae is familly name concatenated with column name with : char in middle. so cannot match V2, AND or OR given the missing :
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49512843
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11412,9 +11924,54 @@ RelExpr * HbaseAccess::preCodeGen(Generator * generator,
}
}
- // add all the key columns. If values are missing in hbase, then atleast the key
+ // add key columns. If values are missing in hbase, then atleast the key
// value is needed to retrieve a row.
- HbaseAccess::addColReferenceFromVIDlist(getIndexDesc()->getIndexKey(), retColRefSet_);
+ //only if needed. If there is already a non nullable non added non nullable with default columns in the set, we should not need to add
+ //any other columns.
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MEDIUM && getMdamKeyPtr() == NULL){ //only enable column retrieval optimization with DF_MEDIUM and not for MDAM scan
+ bool needAddingNonNullableColumn = true; //assume we need to add one non nullable column
+ for (ValueId vid = retColRefSet_.init();// look for each column in th eresult set if one match the criteria non null non added non nullable with default
+ retColRefSet_.next(vid);
+ retColRefSet_.advance(vid))
+ {
+ if (originExePreds->isNotNullable(vid)){// it is non nullable
+ NAColumn * nac;
+ switch (vid.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)vid.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)vid.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
--- End diff --
Doesn't this leave nac uninitialized?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49504311
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
:-) touche
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49485376
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
--- End diff --
I think the following is always better. No?
for (Lng32 i = 0; i < colNameLen; i++)
{
colName[i] = currPtr[currPos++];
}
On Tue, Jan 12, 2016 at 11:06 AM, Eric Owhadi <no...@github.com>
wrote:
> In core/sql/generator/GenExplain.cpp
> <https://github.com/apache/incubator-trafodion/pull/255#discussion_r49482410>
> :
>
> > + listOfColNames->position();
> > + for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
> > + {
> > + char * currPtr = (char*)listOfColNames->getCurr();
> > +
> > + Lng32 currPos = 0;
> > + Lng32 jj = 0;
> > + short colNameLen = *(short*)currPtr;
> > + currPos += sizeof(short);
> > + char colFam[100];
> > + while (currPtr[currPos] != ':')
> > + {
> > + colFam[jj] = currPtr[currPos];
> > + currPos++;
> > + jj++;
> > + }
>
> code compaction vs readability is a long debated topic and end up being
> developer taste specific. I personally prefer compaction too, but wanted to
> respect the model I followed to favor consistancy over my personal taste.
> see the code I refer too:
> for (Lng32 i = 0; i < colNameLen; i++)
> {
> colName[i] = currPtr[currPos];
> currPos++;
> }
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-trafodion/pull/255/files#r49482410>.
>
--
Regards, --Qifan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49412057
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
+ opList_.insert("AND"); // insert an AND node since both side are OK to push down
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we just pushed it down, so increase the node count pushed down.
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (leftOK){ // if only left is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate left tree
+ newExePreds.insert(rightPreds); //make sure we add the right child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we pushed down left side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (rightOK){// if only right is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate right tree
+ newExePreds.insert(leftPreds);//make sure we add the left child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++;// we pushed down right side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else{
+ if(!checkOnly){
+ newExePreds.insert(vid);// we pushed down nothing, make sure the whole node is evaluated by Executor
+ }
+ if (preds.entries()==1)
+ return FALSE;
+
+ }
+ }
+ else{//if not first AND layer, both left and right must be pushable to get anything pushed
+ if(extractHbaseFilterPredsV2(generator, leftPreds, newExePreds, TRUE, FALSE)&&
--- End diff --
Do you think the recursive call is ok? In some cases we may have an OR predicate with thousands of recursive calls. We have run into stack overflows in the past. For the OR case, it may be worth converting the OR backbone into a ValueIdSet or ValueIdList and iterating over that instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49478560
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -358,6 +365,417 @@ void resetAutoFlush() {
table.setAutoFlush(true, true);
}
+ private enum Op {
+ EQUAL, EQUAL_NULL, NOT_EQUAL, NOT_EQUAL_NULL, LESS, LESS_NULL, LESS_OR_EQUAL, LESS_OR_EQUAL_NULL, GREATER, GREATER_NULL,
+ GREATER_OR_EQUAL, GREATER_OR_EQUAL_NULL, NO_OP, NO_OP_NULL,IS_NULL, IS_NULL_NULL, IS_NOT_NULL, IS_NOT_NULL_NULL, AND, OR};
+
+ private Filter SingleColumnValueExcludeOrNotFilter(byte[] columnToFilter,
+ CompareOp op,
+ ByteArrayComparable comparator,
+ HashMap<String,Object> columnsToRemove,
+ Boolean... filterIfMissing){
+ Filter result;
+ boolean fMissing = filterIfMissing.length>0?filterIfMissing[0]:false;//default to false
+ if ((columnsToRemove == null) || !columnsToRemove.containsKey(new String(columnToFilter))){
+ result = new SingleColumnValueFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueFilter)result).setFilterIfMissing(fMissing);
+ }
+ else{
+ result= new SingleColumnValueExcludeFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueExcludeFilter)result).setFilterIfMissing(fMissing);
+ }
+ return result;
+ }
+
+ // construct the hbase filter
+ // optimizes for OR and AND associativity
+ // optimizes for detection of a<? and a>? on nullable and non nullable column equivalent to a<>?
+ // optimize for null check factorization (A not null and (A <op> ?)) or (A not null and A <op2> ?) -> A not null and (A <op> ? or A <op2> ?)
+ // this is an important optimzation for IN statement on non null column
+ // uses the columnToRemove parametter to know if we need to use the SingleColumnValue Exclude or not method to limit returned columns
+
+ private Filter constructV2Filter(Object[] colNamesToFilter,
+ Object[] compareOpList,
+ Object[] colValuesToCompare,
+ HashMap<String,Object> columnsToRemove){
+ LinkedList linkedList = new LinkedList();
+ //populate the list with nodes in reverse polish notation order.
+ int k=0;//column index
+ int kk=0;//value index
--- End diff --
Better name these variables more precisely such as colIndex, valIndex.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49525333
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
in this explicit example, the code is optimized not to use qualifier and family filter, instead it used the SingleColumnValueExcludeFilter, therefore not adding the overhead. Now if instead you do select a,b from t where B>10 and B<20 and c<5, then the optimization is not applicable and yes, I use the qualifier/family filter to pull in a and b only. Note that if a is nullable, and b is nullable, then I will also add one column from the key, avoiding _SALT_ if possible to make sure "absent" null are correctly accounted for. The column to retrieved is computed in HbaseAccess::computeRetrievedCols() that you may not see since it did not change. The trick is that by moving the predicate pushdown computing before invoking computeRetrievedCols in V2, the executorPred() (remaining predicate that cannot be pushed down if any) is already containing nothing if predates will be pushed down. So column computation will not include columns that are needed for predicate evaluation at hbase leve
l. Now for the operator evaluation think you are right. Will have to experiment with it and see where is the point of diminishing return. One thing to note is that this is the last item on the filter list on purpose, so this is not evaluated on filtered out data. Had same concern and tried to put all mitigation possible... This whole thing about limiting data is intuitely attractive, but may be counter productive (CPU intense vs memory/network IO friendly). But given HBase implemented these filter out of the box, I assume they must have seen a benefit...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49467465
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
--- End diff --
It seems we need to specifically test for constants and parameters on child(1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49608162
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
--- End diff --
OK make sense, will add in both V1 and V2. Anyway with all these changes I'll have to re run regress on both V1 and V2, so not shy in changing V1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49255795
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
--- End diff --
Are DATETIMES and INTERVALS all OK?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49360929
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
--- End diff --
What I usually use is snprintf(buf, sizeof(buf), ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49506857
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
--- End diff --
Yes, I have to respond here because this is one of my pet peeves: Making two methods with the same name and different parameter lists is almost always a BAD idea! Doing it like this with a clear indication in the name of what's the difference is much better. Overloading should be used only sparingly, in cases where it really makes sense - IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49477153
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11412,9 +11924,54 @@ RelExpr * HbaseAccess::preCodeGen(Generator * generator,
}
}
- // add all the key columns. If values are missing in hbase, then atleast the key
+ // add key columns. If values are missing in hbase, then atleast the key
// value is needed to retrieve a row.
- HbaseAccess::addColReferenceFromVIDlist(getIndexDesc()->getIndexKey(), retColRefSet_);
+ //only if needed. If there is already a non nullable non added non nullable with default columns in the set, we should not need to add
+ //any other columns.
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MEDIUM && getMdamKeyPtr() == NULL){ //only enable column retrieval optimization with DF_MEDIUM and not for MDAM scan
+ bool needAddingNonNullableColumn = true; //assume we need to add one non nullable column
+ for (ValueId vid = retColRefSet_.init();// look for each column in th eresult set if one match the criteria non null non added non nullable with default
+ retColRefSet_.next(vid);
+ retColRefSet_.advance(vid))
+ {
+ if (originExePreds->isNotNullable(vid)){// it is non nullable
+ NAColumn * nac;
+ switch (vid.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)vid.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)vid.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && !(nac->isAddedColumn() && nac->getDefaultValue())){//check if added and with default... notgood
+ needAddingNonNullableColumn = false; // we found one column meeting all criteria
--- End diff --
Same comment before. this body of logic should be refactored into a method on ValueId and we can that method here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49411109
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
--- End diff --
This is where I should have commented: isFirstAndLayer should never be true when we see an AND, so I would suggest removing this code and also the isFirstAndLayer parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49478186
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
--- End diff --
this is identical to V1 code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49490718
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
--- End diff --
I am sorry my comment was not rendered well due to html poor html conversion:the hasColReference is use on the opposite side of the expression basically to filter out anything like col op function(col), because we only support col op val. So this is correct and this is V1 code too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49255682
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
--- End diff --
Should initialize to null. Otherwise the default case will be taken at line 11425, and God knows what will happen when you reach line 11428 (but in release builds its not likely to be pretty).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49525043
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
+ opList_.insert("AND"); // insert an AND node since both side are OK to push down
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we just pushed it down, so increase the node count pushed down.
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (leftOK){ // if only left is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate left tree
+ newExePreds.insert(rightPreds); //make sure we add the right child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we pushed down left side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (rightOK){// if only right is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate right tree
+ newExePreds.insert(leftPreds);//make sure we add the left child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++;// we pushed down right side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else{
+ if(!checkOnly){
+ newExePreds.insert(vid);// we pushed down nothing, make sure the whole node is evaluated by Executor
+ }
+ if (preds.entries()==1)
+ return FALSE;
+
+ }
+ }
+ else{//if not first AND layer, both left and right must be pushable to get anything pushed
+ if(extractHbaseFilterPredsV2(generator, leftPreds, newExePreds, TRUE, FALSE)&&
--- End diff --
Yes, that all sounds very good to me!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49462141
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
+ ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
+
+ workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
+ .setDataPointer(hbaseFilterValRow_);
+
+ ex_expr::exp_return_type evalRetCode =
+ hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
+ if (evalRetCode == ex_expr::EXPR_ERROR)
+ {
+ return -1;
+ }
+
+ ExpTupleDesc * hfrTD =
+ hbaseAccessTdb().workCriDesc_->getTupleDescriptor
+ (hbaseAccessTdb().hbaseFilterValTuppIndex_);
+
+ hbaseFilterValues_.clear();
+ for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
--- End diff --
Need to add comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49489771
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -754,16 +1017,30 @@ HbaseAccess::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
}
+ // get column retrieved
+ if (((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames()){
+ description += "column_retrieved: ";
+ appendListOfColumns(((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames(),tdb,description);
+ }
+ // get predicate pushed down in Reverse Polish Notation for the AND / OR operators.
+ // could transform it standard notation for better readability, but good enough for now...
+ // could also evaluate the constants instead of hard coded ?, but good enough for now...
+ if (((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames()){
+ description += "pushed_down_rpn: ";
--- End diff --
How about I only change it if I see other changes that needs regenerating EXPECTED? If you feel strong, I'll change it, but letting you know that the impact is more than just one file... many EXPECTED test to generate and change? thought?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49462587
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
--- End diff --
Could use operator ++ here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49468359
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
There is a code duplication "smell" here as well. Please consider some refactoring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49411689
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
--- End diff --
Sorry, I'm not deep enough in the code yet, so I'll just ask: Is hasColReference() a good enough test for HBase filters? In other words, if we have functions like substring(), cos() in the expression, or parameters, are those ok to be evaluated before we make the comparison value for the filter? Probably yes, just checking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49486956
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
--- End diff --
You used this reason of it being in V1 repeatedly, and it's understandable, but I hope you won't repeat errors just because there were there before. Most of our code will show some errors and inefficiencies if we do a review of it. Also, ultimately we should get rid of the V1 code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49525593
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -358,6 +365,417 @@ void resetAutoFlush() {
table.setAutoFlush(true, true);
}
+ private enum Op {
+ EQUAL, EQUAL_NULL, NOT_EQUAL, NOT_EQUAL_NULL, LESS, LESS_NULL, LESS_OR_EQUAL, LESS_OR_EQUAL_NULL, GREATER, GREATER_NULL,
+ GREATER_OR_EQUAL, GREATER_OR_EQUAL_NULL, NO_OP, NO_OP_NULL,IS_NULL, IS_NULL_NULL, IS_NOT_NULL, IS_NOT_NULL_NULL, AND, OR};
+
+ private Filter SingleColumnValueExcludeOrNotFilter(byte[] columnToFilter,
+ CompareOp op,
+ ByteArrayComparable comparator,
+ HashMap<String,Object> columnsToRemove,
+ Boolean... filterIfMissing){
+ Filter result;
+ boolean fMissing = filterIfMissing.length>0?filterIfMissing[0]:false;//default to false
+ if ((columnsToRemove == null) || !columnsToRemove.containsKey(new String(columnToFilter))){
+ result = new SingleColumnValueFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueFilter)result).setFilterIfMissing(fMissing);
+ }
+ else{
+ result= new SingleColumnValueExcludeFilter(getFamily(columnToFilter), getName(columnToFilter), op, comparator);
+ ((SingleColumnValueExcludeFilter)result).setFilterIfMissing(fMissing);
+ }
+ return result;
+ }
+
+ // construct the hbase filter
+ // optimizes for OR and AND associativity
+ // optimizes for detection of a<? and a>? on nullable and non nullable column equivalent to a<>?
+ // optimize for null check factorization (A not null and (A <op> ?)) or (A not null and A <op2> ?) -> A not null and (A <op> ? or A <op2> ?)
+ // this is an important optimzation for IN statement on non null column
+ // uses the columnToRemove parametter to know if we need to use the SingleColumnValue Exclude or not method to limit returned columns
+
+ private Filter constructV2Filter(Object[] colNamesToFilter,
+ Object[] compareOpList,
+ Object[] colValuesToCompare,
+ HashMap<String,Object> columnsToRemove){
+ LinkedList linkedList = new LinkedList();
+ //populate the list with nodes in reverse polish notation order.
+ int k=0;//column index
+ int kk=0;//value index
+ for (int i=1; i<compareOpList.length; i++){ // skip first one containing "V2" marker
+ String opStr = new String((byte[])compareOpList[i]);
+ switch(Op.valueOf(opStr)){
+
+ case EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove
+ ));
+ k++;kk++;
+ break;
+ case EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true //filterIfMissing
+ ),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case NOT_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case NOT_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case LESS:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case LESS_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case LESS_OR_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case LESS_OR_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.LESS_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case GREATER:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case GREATER_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case GREATER_OR_EQUAL:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case GREATER_OR_EQUAL_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.GREATER_OR_EQUAL,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case NO_OP:
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove));
+ k++;kk++;
+ break;
+ case NO_OP_NULL:
+ linkedList.addLast(new FilterList(FilterList.Operator.MUST_PASS_ALL, //AND between if not null and the actual
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{0x00}),//check for null indicator = 0 representing non null
+ columnsToRemove,
+ true), //filterIfMissing,
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP,
+ new BinaryComparator((byte[])colValuesToCompare[kk]),
+ columnsToRemove)));
+ k++;kk++;
+ break;
+ case IS_NULL:
+ // is null on a non nullable column!
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NO_OP, //exclude everything
+ new BinaryPrefixComparator((new byte[]{})),
+ columnsToRemove));
+ k++;
+ break;
+ case IS_NULL_NULL:
+ // is_null on nullable column: is absent OR has the first byte set to FF indicating NULL.
+ linkedList.addLast(
+ new FilterList(FilterList.Operator.MUST_PASS_ONE, //OR
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new NullComparator(),//is absent?
+ columnsToRemove),
+ SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.EQUAL,
+ new BinaryPrefixComparator(new byte[]{-1}),//0xFF has null prefix indicator
+ columnsToRemove)));
+ k++;
+ break;
+ case IS_NOT_NULL:
+ // is not null on a non nullable column!
+ // do nothing, always true
+ k++;
+ break;
+ case IS_NOT_NULL_NULL:
+ // is_not_null on nullable column: is not absent AND has the first byte not set to FF indicating NULL.
+ linkedList.addLast(SingleColumnValueExcludeOrNotFilter(
+ (byte[])colNamesToFilter[k],
+ CompareOp.NOT_EQUAL,
+ new BinaryPrefixComparator(new byte[]{-1}),// 0xFF has null prefix indicator
+ columnsToRemove,
+ true));//filter if missing (if absent null)
+ k++;
+ break;
+ case AND:
+ linkedList.addLast("AND");
+ break;
+ case OR:
+ linkedList.addLast("OR");
+ break;
+ default:
+ }//switch
+ }//for
--- End diff --
in the java code constructV2Filter is where RPN is evaluated and turned into hbase filter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49514116
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
--- End diff --
I can't really see how it could be split into a checking and an executing method and how that would help. Unless I see a nice alternative I think I'm fine with this single method, and deleting the top-level AND code will also make the method less complex.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49227654
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -2808,53 +2808,51 @@ short ExHbaseAccessTcb::setupHbaseFilterPreds()
(hbaseAccessTdb().listOfHbaseFilterColNames()->numEntries() == 0))
return 0;
- if (! hbaseFilterValExpr())
- return 0;
-
- ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
-
- workAtp_->getTupp(hbaseAccessTdb().hbaseFilterValTuppIndex_)
- .setDataPointer(hbaseFilterValRow_);
-
- ex_expr::exp_return_type evalRetCode =
- hbaseFilterValExpr()->eval(pentry_down->getAtp(), workAtp_);
- if (evalRetCode == ex_expr::EXPR_ERROR)
- {
- return -1;
- }
-
- ExpTupleDesc * hfrTD =
- hbaseAccessTdb().workCriDesc_->getTupleDescriptor
- (hbaseAccessTdb().hbaseFilterValTuppIndex_);
-
- hbaseFilterValues_.clear();
- for (Lng32 i = 0; i < hfrTD->numAttrs(); i++)
- {
- Attributes * attr = hfrTD->getAttr(i);
-
- if (attr)
- {
- NAString value(getHeap());
- if (attr->getNullFlag())
- {
- char nullValChar = 0;
-
- short nullVal = *(short*)&hbaseFilterValRow_[attr->getNullIndOffset()];
-
- if (nullVal)
- nullValChar = -1;
- value.append((char*)&nullValChar, sizeof(char));
- }
-
- char * colVal = &hbaseFilterValRow_[attr->getOffset()];
-
- value.append(colVal,
- attr->getLength(&hbaseFilterValRow_[attr->getVCLenIndOffset()]));
-
- hbaseFilterValues_.insert(value);
- }
- }
-
+ if (hbaseFilterValExpr()){// with pushdown V2 it can be null if we have only unary operation
--- End diff --
are you seeing the problem only on the new code, or was it already in the old one? Because I do see bad indentation in one or 2 area, but that is old code, nothing added?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49465058
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
--- End diff --
the same code to identify ":" and copy duplicated here. Please code it into a function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49505528
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
--- End diff --
Sounds good, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49369767
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
Same as above, limit taken from existing code. same 3 options:in ComTdbHbaseAccess.cpp:
static void showColNames(Queue * listOfColNames, Space * space)
{
char buf[1000];
listOfColNames->position();
for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
{
char * currPtr = (char*)listOfColNames->getCurr();
Lng32 currPos = 0;
Lng32 jj = 0;
short colNameLen = *(short*)currPtr;
currPos += sizeof(short);
char colFam[100];
while (currPtr[currPos] != ':')
{
colFam[jj] = currPtr[currPos];
currPos++;
jj++;
}
colFam[jj] = ':';
jj++;
currPos++;
colFam[jj] = 0;
colNameLen -= jj;
NABoolean withAt = FALSE;
char * colName = &currPtr[currPos];
if (colName[0] == '@')
{
colNameLen--;
colName++;
withAt = TRUE;
}
Int64 v;
if (colNameLen == sizeof(char))
v = *(char*)colName;
else if (colNameLen == sizeof(unsigned short))
v = *(UInt16*)colName;
else if (colNameLen == sizeof(Lng32))
v = *(ULng32*)colName;
else
v = 0;
str_sprintf(buf, " Entry #%d: %s%s%Ld",
j+1,
colFam,
(withAt ? "@" : ""),
v);
space->allocateAndCopyToAlignedSpace(buf, str_len(buf), sizeof(short));
listOfColNames->advance();
} // for
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49477898
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
this is V1 code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49517247
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
+ opList_.insert("AND"); // insert an AND node since both side are OK to push down
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we just pushed it down, so increase the node count pushed down.
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (leftOK){ // if only left is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate left tree
+ newExePreds.insert(rightPreds); //make sure we add the right child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++; // we pushed down left side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else if (rightOK){// if only right is OK to push down
+ if(!checkOnly){
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate right tree
+ newExePreds.insert(leftPreds);//make sure we add the left child to predicates that needs executor evalvaluation
+ if (addedNode>0)opList_.insert("AND"); // if it is not the first node addd to the push down, AND it with the rest
+ addedNode++;// we pushed down right side so mark it
+ }
+ if (preds.entries()==1)
+ return TRUE;
+ } else{
+ if(!checkOnly){
+ newExePreds.insert(vid);// we pushed down nothing, make sure the whole node is evaluated by Executor
+ }
+ if (preds.entries()==1)
+ return FALSE;
+
+ }
+ }
+ else{//if not first AND layer, both left and right must be pushable to get anything pushed
+ if(extractHbaseFilterPredsV2(generator, leftPreds, newExePreds, TRUE, FALSE)&&
--- End diff --
cool, I am thinking adding a CQD to set the MAX OR size hbase push down and default it to conservative 10 until we have more test data to know when it is good? Thought? In a new version we could think about pushing down MIN MAX, and keep the or list at executor level, putting back the column in question in the result set so it can be re evaluated by executor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49606833
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
+ {
+ colName[i] = currPtr[currPos];
+ currPos++;
+ }
+
+ colName[colNameLen] = 0;
+
+ colNamePtr = colName;
+
+ if (j==0)
+ str_sprintf(buf, "%s",colNamePtr);
+ else
+ str_sprintf(buf, ",%s",colNamePtr);
+
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+
+ }// hbase native table
+ outNAString +=" ";
+}
+
+static void appendPushedDownExpression(ComTdb *tdb, NAString& outNAString){
+ // in predicate pushdown V2, the hbaseCompareOps list contains a reverse polish set of operation, were operators are
+ // AND or OR, the rest are operands. this function display the column, operator and replace any constant with ?. it keeps reverse polish format
+ // this can be improved in the future for better readability.
+ char buf[1000];
+ Queue* reversePolishItems = ((ComTdbHbaseAccess *)tdb)->listOfHbaseCompareOps();
+ Queue* pushedDownColumns = ((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames();
+ reversePolishItems->position();
+ pushedDownColumns->position();
+
+ for (Lng32 j = 0; j < reversePolishItems->numEntries(); j++){
+ char * currPtr = (char*)reversePolishItems->getCurr();
+ char buf2[1000];
+ if (strcmp(currPtr,"V2")!=0 && strcmp(currPtr,"AND")!=0 && strcmp(currPtr,"OR")!=0){//if an operand (not an operator or V2 marker), get the column name
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char * currPtr2 = (char*)pushedDownColumns->getCurr();
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr2;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr2[currPos] != ':')
--- End diff --
opinion on this Hand and Dave, I don't mind changing it if most people think it should?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49607785
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -754,16 +1017,30 @@ HbaseAccess::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
}
+ // get column retrieved
+ if (((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames()){
+ description += "column_retrieved: ";
+ appendListOfColumns(((ComTdbHbaseAccess *)tdb)->listOfFetchedColNames(),tdb,description);
+ }
+ // get predicate pushed down in Reverse Polish Notation for the AND / OR operators.
+ // could transform it standard notation for better readability, but good enough for now...
+ // could also evaluate the constants instead of hard coded ?, but good enough for now...
+ if (((ComTdbHbaseAccess *)tdb)->listOfHbaseFilterColNames()){
+ description += "pushed_down_rpn: ";
--- End diff --
OK clear, I will change it. Thanks for caching this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49254252
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
--- End diff --
Does Trafodion impose a limit on column family name length? Could we get an overrun here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49473749
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
--- End diff --
Same question as putting V2 into the opList_. This function has two more arguments than the v1 version. So there is no confusion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49373698
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
--- End diff --
Let me research that, the V2 code is identical to V1. So if V1 was correct V2 is. I must admit I did not check for V1 correctness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49475250
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
+ extractHbaseFilterPredsV2(generator,preds,newExePreds,FALSE,TRUE);
+ return 0;
+
+}
+
+// return true if successfull push down of node
+NABoolean HbaseAccess::extractHbaseFilterPredsV2(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds, NABoolean checkOnly, NABoolean isFirstAndLayer )
+{
+
+ // the isFirstAndLayer is used to allow detecting top level predicate that can still be pushed to executor
+ int addedNode=0;
+ for (ValueId vid = preds.init();
+ (preds.next(vid));
+ preds.advance(vid))
+ {
+ ItemExpr * ie = vid.getItemExpr();
+
+ // if it is AND operation, recurse through left and right children
+ if (ie->getOperatorType() == ITM_AND)
+ {
+ ValueIdSet leftPreds;
+ ValueIdSet rightPreds;
+ leftPreds += ie->child(0)->castToItemExpr()->getValueId();
+ rightPreds += ie->child(1)->castToItemExpr()->getValueId();
+ if (isFirstAndLayer){
+ NABoolean leftOK = extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,TRUE, TRUE);
+ NABoolean rightOK = extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,TRUE, TRUE);
+ if (leftOK && rightOK){
+ if (!checkOnly){
+ extractHbaseFilterPredsV2(generator, leftPreds, newExePreds,FALSE, TRUE);//generate tree
+ extractHbaseFilterPredsV2(generator, rightPreds, newExePreds,FALSE, TRUE);//generate tree
--- End diff --
Note that in general a function should be coded to do one thing only. Is it possible to separate the check functionality from the extraction functionality in extractHbaseFilterPredsV2(), so that the logic can be more cleanly stated as
if (isFeasibleToPerformHbasePushdown())
extractHbasePushdownPredicates()
The compiler has many instances of coding patterns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49488177
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(newCVV);
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+ valueVID = newCV->getValueId();
+ }
+
+ ItemExpr * castValue = NULL;
+ if (NOT hbaseLookupPred)
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &colType);
+ else
+ {
+ castValue = new(generator->wHeap()) Cast(valueVID.getItemExpr(), &valueVID.getType());
+ }
+
+ if ((NOT hbaseLookupPred) &&
+ (isEncodingNeededForSerialization(colVID.getItemExpr())))
+ {
+ castValue = new(generator->wHeap()) CompEncode
+ (castValue, FALSE, -1, CollationInfo::Sort, TRUE, FALSE);
+ }
+
+ castValue = castValue->bindNode(generator->getBindWA());
+ castValue = castValue->preCodeGen(generator);
+
+ valueVID = castValue->getValueId();
+
+ NAString nullType;
+
+ if ((colType.supportsSQLnull()) ||
+ (valueType.supportsSQLnull()))
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+
+ // append -NULL to the operator to signify the java code generating pushdown filters to handle NULL semantic logic
+ if (ie->getOperatorType() == ITM_EQUAL)
+ op = "EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_NOT_EQUAL)
+ op = "NOT_EQUAL"+nullType;
+ else if (ie->getOperatorType() == ITM_LESS){
+ if (flipOp)
+ op = "GREATER"+nullType;
+ else
+ op = "LESS"+nullType;
+ }
+ else if (ie->getOperatorType() == ITM_LESS_EQ){
+ if (flipOp)
+ op = "GREATER_OR_EQUAL"+nullType;
+ else
+ op = "LESS_OR_EQUAL"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER){
+ if (flipOp)
+ op = "LESS"+nullType;
+ else
+ op = "GREATER"+nullType;
+ }else if (ie->getOperatorType() == ITM_GREATER_EQ){
+ if (flipOp)
+ op = "LESS_OR_EQUAL"+nullType;
+ else
+ op = "GREATER_OR_EQUAL"+nullType;
+ }else
+ op = "NO_OP"+nullType;
+ }
+ }
+ if (foundUnary){
+ const NAType &colType = colVID.getType();
+ NAString nullType;
+
+ if (colType.supportsSQLnull())
+ {
+ nullType = "_NULL";
+ }
+ else
+ {
+ nullType = "";
+ }
+ if (ie->getOperatorType() == ITM_IS_NULL)
+ op = "IS_NULL"+nullType;
+ else if (ie->getOperatorType() == ITM_IS_NOT_NULL)
+ op = "IS_NOT_NULL"+nullType;
+ }
+
+ return foundBinary || foundUnary;
+}
+short HbaseAccess::extractHbaseFilterPredsVX(Generator * generator,
+ ValueIdSet &preds, ValueIdSet &newExePreds){
+ //separate the code that should not belong in the recursive function
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_OFF)
+ return 0;
+ // check if initial (version 1) implementation
+ if (CmpCommon::getDefault(HBASE_FILTER_PREDS) == DF_MINIMUM)
+ return extractHbaseFilterPreds(generator,preds,newExePreds);
+
+ // if here, we are DF_MEDIUM
+ // cannot push preds for aligned format row
+ NABoolean isAlignedFormat = getTableDesc()->getNATable()->isAlignedFormat(getIndexDesc());
+
+ if (isAlignedFormat)
+ return 0;
+ //recursive function call
+ opList_.insert("V2");//to instruct the java side that we are dealing with predicate pushdown V2 semantic, add "V2" marker
--- End diff --
I saw the V2 token is removed in the Java code. The very fact that pushdown is happening with V2 can be detected by the explain output.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by eowhadi <gi...@git.apache.org>.
Github user eowhadi commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49616154
--- Diff: core/sql/src/main/java/org/trafodion/sql/HTableClient.java ---
@@ -421,29 +839,140 @@ else if (versions > 0)
else
numColsInScan = 0;
if (colNamesToFilter != null) {
- FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
-
- for (int i = 0; i < colNamesToFilter.length; i++) {
- byte[] colName = (byte[])colNamesToFilter[i];
- byte[] coByte = (byte[])compareOpList[i];
- byte[] colVal = (byte[])colValuesToCompare[i];
-
- if ((coByte == null) || (colVal == null)) {
- return false;
- }
-
- String coStr = new String(coByte);
- CompareOp co = CompareOp.valueOf(coStr);
-
- SingleColumnValueFilter filter1 =
- new SingleColumnValueFilter(getFamily(colName), getName(colName),
- co, colVal);
- list.addFilter(filter1);
- }
-
+ FilterList list;
+ boolean narrowDownResultColumns = false; //to check if we need a narrow down column filter (V2 only feature)
+ if (compareOpList == null)return false;
+ if (new String((byte[])compareOpList[0]).equals("V2")){ // are we dealing with predicate pushdown V2
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ HashMap<String,Object> columnsToRemove = new HashMap<String,Object>();
+ //if columnsToRemove not null, we are narrowing down using the SingleColumnValue[Exclude]Filter method
+ //else we will use the explicit FamilyFilter and QualifierFilter
+ //the simplified logic is that we can use the first method if and only if each and every column in the
+ //pushed down predicate shows up only once.
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+
+ // check if the filter column is already part of the column list, if not add it if we are limiting columns (not *)
+ if(columns!=null && columns.length > 0){// if not *
+ boolean columnAlreadyIn = false; //assume column not yet in the scan object
+ for (int k=0; k<columns.length;k++){
+ if (Arrays.equals(colName, (byte[])columns[k])){
+ columnAlreadyIn = true;//found already exist
+ break;//no need to look further
+ }
+ }
+ if (!columnAlreadyIn){// column was not already in, so add it
+ scan.addColumn(getFamily(colName),getName(colName));
+ narrowDownResultColumns = true; //since we added a column for predicate eval, we need to remove it later out of result set
+ String strColName = new String(colName);
+ if (columnsToRemove != null && columnsToRemove.containsKey(strColName)){// if we already added this column, it means it shows up more than once
+ columnsToRemove = null; // therefore, use the FamilyFilter/QualifierFilter method
+ }else if (columnsToRemove != null)// else
+ columnsToRemove.put(strColName,null); // add it to the list of column that should be nuked with the Exclude version of the SingleColumnValueFilter
+ }
+ }
+ }
+ if (columnsToRemove != null)
+ { //we are almost done checking if Exclude version of SingleColumnnValueFilter can be used. Th elast check s about to know if there is a IS_NULL_NULL
+ //operation that cannot be using the Exclude method, as it is transformed in a filterList with OR, therefore we cannot guaranty that the SingleColumnValueExcludeFilter
+ //performing the exclusion will be reached.
+ boolean is_null_nullFound = false;
+ for (Object o:compareOpList ){
+ if (new String((byte[])o).equals("IS_NULL_NULL")){
+ is_null_nullFound = true;
+ break;
+ }
+ }
+ if (is_null_nullFound){
+ columnsToRemove = null; // disable Exclude method version of SingleColumnnValueFilter
+ }else
+ narrowDownResultColumns = false; // we will use the Exclude version of SingleColumnnValueFilter, so bypass the Family/QualifierFilter method
+ }
+ Filter f =constructV2Filter(colNamesToFilter,compareOpList,colValuesToCompare, columnsToRemove);
+ if (f==null) return false; // error logging done inside constructV2Filter
+ list.addFilter(f);
+ }//end V2
+ else{// deal with V1
+ list = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+
+ for (int i = 0; i < colNamesToFilter.length; i++) {
+ byte[] colName = (byte[])colNamesToFilter[i];
+ byte[] coByte = (byte[])compareOpList[i];
+ byte[] colVal = (byte[])colValuesToCompare[i];
+
+ if ((coByte == null) || (colVal == null)) {
+ return false;
+ }
+ String coStr = new String(coByte);
+ CompareOp co = CompareOp.valueOf(coStr);
+
+ SingleColumnValueFilter filter1 =
+ new SingleColumnValueFilter(getFamily(colName), getName(colName),
+ co, colVal);
+ list.addFilter(filter1);
+ }
+ }//end V1
+ // if we added a column for predicate eval, we need to filter down result columns
+ FilterList resultColumnsOnlyFilter = null;
+ if (narrowDownResultColumns){
+ HashMap<String,ArrayList<byte[]>> hm = new HashMap<String,ArrayList<byte[]>>(3);//use to deal with multiple family table
+ // initialize hm with list of columns requested for output
+ for (int i=0; i<columns.length; i++){ // if we are here we know columns is not null
+ if (hm.containsKey(new String(getFamily((byte[])columns[i])))){
+ hm.get(new String(getFamily((byte[])columns[i]))).add((byte[])columns[i]);
+ }else{
+ ArrayList<byte[]> al = new ArrayList<byte[]>();
+ al.add((byte[])columns[i]);
+ hm.put(new String(getFamily((byte[])columns[i])), al);
+ }
+ }
+
+ if (hm.size()==1){//only one column family
+ resultColumnsOnlyFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
--- End diff --
I reviewed the code and it uses the principles I was going to use to implement the separation if people agree that this is the way to go (adding Hans removal of first level AND). But I still feel this is not the right move. This trigger similar code to be duplicated in 2 function with risk of missing synchronizing any changes. Hans / Dave do you feel any different now that you see how splitting would look like? I am ready to do ether way...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by zellerh <gi...@git.apache.org>.
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49828660
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -11274,6 +11274,485 @@ short HbaseAccess::extractHbaseFilterPreds(Generator * generator,
return 0;
}
+////////////////////////////////////////////////////////////////////////////
+// To push down, the predicate must have the following form:
+// xp:= <column> <op> <value-expr>
+// xp:= <column> is not null (no support for hbase lookup)
+// xp:= <column> is null (no support for hbase lookup)
+// (xp:=<column> like <value-expr> not yet implemented)
+// xp:=<xp> OR <xp> (not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+// xp:=<xp> AND <xp>(not evaluated in isHbaseFilterPredV2, but by extractHbaseFilterPredV2)
+//
+// and all of the following conditions must be met:
+//
+// <column>: a base table or index column which can be serialized and belong to the table being scanned.
+// serialized: either the column doesn't need encoding, like
+// an unsigned integer, or the column
+// was declared with the SERIALIZED option.
+// it also must not be an added column with default non null.
+// <op>: eq, ne, gt, ge, lt, le
+// <value-expr>: an expression that only contains const or param values, and
+// <value-expr>'s datatype is not a superset of <column>'s datatype.
+//
+/////////////////////////////////////////////////////////////////////////////
+NABoolean HbaseAccess::isHbaseFilterPredV2(Generator * generator, ItemExpr * ie,
+ ValueId &colVID, ValueId &valueVID,
+ NAString &op)
+{
+ NABoolean foundBinary = FALSE;
+ NABoolean foundUnary = FALSE;
+ NABoolean hbaseLookupPred = FALSE;
+ NABoolean flipOp = FALSE; // set to TRUE when column is child(1)
+
+ if (ie &&
+ ((ie->getOperatorType() >= ITM_EQUAL) &&
+ (ie->getOperatorType() <= ITM_GREATER_EQ))) //binary operator case
+ {//begin expression
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ ItemExpr * child1 = ie->child(1)->castToItemExpr();
+
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_BASECOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_INDEXCOLUMN) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ foundBinary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_REFERENCE) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = ie->child(1)->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ else if ((ie->child(0)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(1))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(0)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(1)->getValueId();
+ }
+ }
+ else if ((ie->child(1)->getOperatorType() == ITM_HBASE_COLUMN_LOOKUP) &&
+ (NOT hasColReference(ie->child(0))))
+ {
+ HbaseColumnLookup * hcl = (HbaseColumnLookup*)ie->child(1)->castToItemExpr();
+ if (hcl->getValueId().getType().getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ hbaseLookupPred = TRUE;
+
+ ItemExpr * newCV = new(generator->wHeap()) ConstValue(hcl->hbaseCol());
+ newCV = newCV->bindNode(generator->getBindWA());
+ newCV = newCV->preCodeGen(generator);
+
+ foundBinary = TRUE;
+ flipOp = TRUE;
+ colVID = newCV->getValueId();
+ valueVID = ie->child(0)->getValueId();
+ }
+ }
+ }//end binary operators
+ else if (ie && ((ie->getOperatorType() == ITM_IS_NULL)||(ie->getOperatorType() == ITM_IS_NOT_NULL))){//check for unary operators
+ ItemExpr * child0 = ie->child(0)->castToItemExpr();
+ if ((ie->child(0)->getOperatorType() == ITM_BASECOLUMN) ||
+ (ie->child(0)->getOperatorType() == ITM_INDEXCOLUMN)||
+ (ie->child(0)->getOperatorType() == ITM_REFERENCE)){
+ foundUnary = TRUE;
+ colVID = ie->child(0)->getValueId();
+ valueVID = NULL_VALUE_ID;
+ }
+
+ }//end unary operators
+
+ //check if found columns belong to table being scanned (so is not an input to the scan node)
+ if (foundBinary || foundUnary){
+ ValueId dummyValueId;
+ if (getGroupAttr()->getCharacteristicInputs().referencesTheGivenValue(colVID,dummyValueId)){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+ //check if not an added column with default non null
+ if ((foundBinary || foundUnary)&& (NOT hbaseLookupPred)){
+ NAColumn * nac;
+ switch (colVID.getItemExpr()->getOperatorType()){
+ case ITM_BASECOLUMN:
+ nac = ((BaseColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ case ITM_INDEXCOLUMN:
+ nac = ((IndexColumn*)colVID.getItemExpr())->getNAColumn();
+ break;
+ default:
+ break;
+ }
+ if (nac && nac->isAddedColumn() && nac->getDefaultValue()){
+ foundBinary=FALSE;
+ foundUnary=FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ const NAType &colType = colVID.getType();
+ const NAType &valueType = valueVID.getType();
+
+ NABoolean generateNarrow = FALSE;
+ if (NOT hbaseLookupPred)
+ {
+ generateNarrow = valueType.errorsCanOccur(colType);
+ if ((generateNarrow) || // value not a superset of column
+ (NOT columnEnabledForSerialization(colVID.getItemExpr())))
+ foundBinary = FALSE;
+ }
+
+ if (foundBinary)
+ {
+ if (colType.getTypeQualifier() == NA_CHARACTER_TYPE)
+ {
+ const CharType &charColType = (CharType&)colType;
+ const CharType &charValType = (CharType&)valueType;
+
+ if ((charColType.isCaseinsensitive() || charValType.isCaseinsensitive()) ||
+ (charColType.isUpshifted() || charValType.isUpshifted()))
+ foundBinary = FALSE;
+ }
+ else if (colType.getTypeQualifier() == NA_NUMERIC_TYPE)
+ {
+ const NumericType &numType = (NumericType&)colType;
+ const NumericType &valType = (NumericType&)valueType;
+ if (numType.isBigNum() || valType.isBigNum())
+ foundBinary = FALSE;
+ }
+ }
+
+ if (foundBinary)
+ {
+ if ((ie) && (((BiRelat*)ie)->addedForLikePred()) &&
+ (valueVID.getItemExpr()->getOperatorType() == ITM_CONSTANT))
+ {
+ // remove trailing '\0' characters since this is being pushed down to hbase.
+ ConstValue * cv = (ConstValue*)(valueVID.getItemExpr());
+ char * cvv = (char*)cv->getConstValue();
+ Lng32 len = cv->getStorageSize() - 1;
+ while ((len > 0) && (cvv[len] == '\0'))
+ len--;
+
+ NAString newCVV(cvv, len+1);
--- End diff --
Yes, thank you for researching and explaining that. Assuming that people won't use a LIKE predicate on binary data is hopefully ok. So I hope this is another one off the list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1662] Predicate push ...
Posted by nonstop-qfchen <gi...@git.apache.org>.
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/255#discussion_r49463378
--- Diff: core/sql/generator/GenExplain.cpp ---
@@ -660,6 +660,269 @@ FileScan::addSpecificExplainInfo(ExplainTupleMaster *explainTuple,
return(explainTuple);
}
+static void appendListOfColumns(Queue* listOfColNames,ComTdb *tdb, NAString& outNAString){
+
+ if (((ComTdbHbaseAccess*)tdb)->sqHbaseTable()){// if trafodion table
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ Lng32 currPos = 0;
+ Lng32 jj = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colFam[100];
+ while (currPtr[currPos] != ':')
+ {
+ colFam[jj] = currPtr[currPos];
+ currPos++;
+ jj++;
+ }
+ colFam[jj] = ':';
+ jj++;
+ currPos++;
+ colFam[jj] = 0;
+ colNameLen -= jj;
+
+ NABoolean withAt = FALSE;
+ char * colName = &currPtr[currPos];
+ if (colName[0] == '@')
+ {
+ colNameLen--;
+ colName++;
+ withAt = TRUE;
+ }
+
+ Int64 v;
+ if (colNameLen == sizeof(char))
+ v = *(char*)colName;
+ else if (colNameLen == sizeof(unsigned short))
+ v = *(UInt16*)colName;
+ else if (colNameLen == sizeof(Lng32))
+ v = *(ULng32*)colName;
+ else
+ v = 0;
+ if (j==0)
+ str_sprintf(buf, "%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+ else
+ str_sprintf(buf, ",%s%s%Ld",
+ colFam,
+ (withAt ? "@" : ""),
+ v);
+
+ outNAString += buf;
+
+ listOfColNames->advance();
+ } // for
+ }// trafodion tables
+ else
+ {// if hbase native tables
+ char buf[1000];
+
+ listOfColNames->position();
+ for (Lng32 j = 0; j < listOfColNames->numEntries(); j++)
+ {
+ char * currPtr = (char*)listOfColNames->getCurr();
+
+ char * colNamePtr = NULL;
+
+ Lng32 currPos = 0;
+ short colNameLen = *(short*)currPtr;
+ currPos += sizeof(short);
+ char colName[500];
+
+ for (Lng32 i = 0; i < colNameLen; i++)
--- End diff --
could we use some form of system string copy functions here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---