You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by js...@apache.org on 2015/04/06 23:28:53 UTC
[3/3] drill git commit: DRILL-2060: Constant folding rule
DRILL-2060: Constant folding rule
2060 update - Constant folding work completed.
Fix issue with date, time and timestamp literal creation.
Fix literal creation during expression interpretation to match nullability of incoming expression.
Fix decimal literals in interpreted expression eval.
Disable test with an exposed planning bug when the project instance of the constant folding rule is enabled. The rule is not actually influencing the final plan when the rule is firing and making expression reductions. This is due to our current cost model fro project which just counts the number of expressions and does not consider expression complexity. The issues have been logged in DRILL-2218 for further investigation, they do not need to be solved to merge the other constant folding rules and all of the interpreted expression work that has been done.
Get rid of clutter in RuleSets, explanation has been moved to the 2218 JIRA.
Belongs with 2060, fix constant expression executor to use the new constant expression interpreter interface that returns a ValueHolder instead a ValueVector with a single value filled in.
2060 update - change test baseline due to new column ordering (no functional or performance impacting changes to plan)
2060 - address Aman's comments.
add test ignore - DRILL-2218
Baseline update for project pushdown test (only column ordering on a scan, no functional or performance impacting plan changes)
Turn back on project instance.
Small casting bug in constant executor.
Don't fold hive UDFs.
Modify DrillBuf to allow a BufferManager to be the owning context for a DrilllBuf.
TODO - refactor to remove remaining common code from OperatorContext and FragmentContext,
have them both use the new BufferManager.
Add system option for disabling constant folding.
2060 update - test option to disable constant folding.
Update RuleSets to actually allow turning the constant folding rules on and off as well as establish general pattern for turning logical rules on an off, similar to how some physical rules can be already.
Change the estimated row count in EasyGroupScan to report a number of files in the case where the file size indicates an estimated total count of 0 records. Allows very small files to be pruned.
Fix folding expressions that result in null after refactoring the interpreted expression evaluation to return a ValueHolder in the case of a constant expression. Previously a value vector was returned in the same manner as the interpreter can still do when given an input VectorAccessible and an expression that may contain fild references. Calling getObject on the output vector previously gracefully handled nulls as they were passed into the Calciate API to create literals. This process has to be a bit more manual now.
Address Jinfeng's review comments.
A few more review comments.
Disable cost calculation change, complete fix will come in 2553.
Throw a runtime exception of there is an error materializing the expression, as the same materialization will take place at query execution time we should fail early.
Add a test that does prune appropriately, still have a test for the outstanding issue tracked in DRILL-2553.
Small fix for test to properly set session option and set it back after completion.
Fixing comment that was garbled somehow.
small fix for case where expression returns a null result during constant folding.
Add a little defensive code to give a good error message if a type that does not appear in the mapping from Drill to Calcite types attempts to be folded into a null value.
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/8d70709d
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/8d70709d
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/8d70709d
Branch: refs/heads/master
Commit: 8d70709ddaddc7111517f4bed883fc9908ef94de
Parents: a53e123
Author: Jason Altekruse <al...@gmail.com>
Authored: Mon Mar 9 11:51:36 2015 -0700
Committer: Jason Altekruse <al...@gmail.com>
Committed: Mon Apr 6 08:13:26 2015 -0700
----------------------------------------------------------------------
.../expression/ExpressionStringBuilder.java | 12 +-
.../drill/exec/planner/sql/HiveUDFOperator.java | 8 +
.../src/main/codegen/templates/TypeHelper.java | 5 +
.../src/main/java/io/netty/buffer/DrillBuf.java | 15 +-
.../exec/expr/ExpressionTreeMaterializer.java | 3 +-
.../exec/expr/fn/DrillFunctionRegistry.java | 13 +-
.../expr/fn/impl/StringFunctionHelpers.java | 15 +
.../fn/interpreter/InterpreterEvaluator.java | 2 +-
.../apache/drill/exec/ops/BufferManager.java | 1 +
.../impl/project/ProjectRecordBatch.java | 3 +-
.../planner/logical/DrillConstExecutor.java | 311 +++++++++++++++++++
.../logical/DrillReduceExpressionsRule.java | 83 +++++
.../exec/planner/logical/DrillRuleSets.java | 64 +++-
.../exec/planner/physical/PlannerSettings.java | 5 +
.../drill/exec/planner/sql/DrillSqlWorker.java | 7 +-
.../server/options/SystemOptionManager.java | 1 +
.../java/org/apache/drill/PlanningBase.java | 10 +-
.../org/apache/drill/TestProjectPushDown.java | 4 +-
.../exec/fn/impl/TestAggregateFunctions.java | 2 +
.../exec/fn/interp/TestConstantFolding.java | 187 +++++++++++
.../src/test/resources/parquet/alltypes.json | 8 +
pom.xml | 2 +-
22 files changed, 742 insertions(+), 19 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/common/src/main/java/org/apache/drill/common/expression/ExpressionStringBuilder.java
----------------------------------------------------------------------
diff --git a/common/src/main/java/org/apache/drill/common/expression/ExpressionStringBuilder.java b/common/src/main/java/org/apache/drill/common/expression/ExpressionStringBuilder.java
index d3839ed..9791d39 100644
--- a/common/src/main/java/org/apache/drill/common/expression/ExpressionStringBuilder.java
+++ b/common/src/main/java/org/apache/drill/common/expression/ExpressionStringBuilder.java
@@ -79,7 +79,17 @@ public class ExpressionStringBuilder extends AbstractExprVisitor<Void, StringBui
@Override
public Void visitFunctionHolderExpression(FunctionHolderExpression holder, StringBuilder sb) throws RuntimeException {
- throw new UnsupportedOperationException();
+ ImmutableList<LogicalExpression> args = holder.args;
+ sb.append(holder.getName());
+ sb.append("(");
+ for (int i = 0; i < args.size(); i++) {
+ if (i != 0) {
+ sb.append(", ");
+ }
+ args.get(i).accept(this, sb);
+ }
+ sb.append(") ");
+ return null;
}
@Override
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HiveUDFOperator.java
----------------------------------------------------------------------
diff --git a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HiveUDFOperator.java b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HiveUDFOperator.java
index 7524690..8a54bfe 100644
--- a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HiveUDFOperator.java
+++ b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/planner/sql/HiveUDFOperator.java
@@ -43,6 +43,14 @@ public class HiveUDFOperator extends SqlFunction {
SqlFunctionCategory.USER_DEFINED_FUNCTION);
}
+ // Consider Hive functions to be non-deterministic so they are not folded at
+ // planning time. The expression interpreter used to evaluate constant expressions
+ // currently does not support anything but simple DrillFuncs.
+ @Override
+ public boolean isDeterministic() {
+ return false;
+ }
+
@Override
public RelDataType deriveType(SqlValidator validator, SqlValidatorScope scope, SqlCall call) {
RelDataTypeFactory factory = validator.getTypeFactory();
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/codegen/templates/TypeHelper.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/codegen/templates/TypeHelper.java b/exec/java-exec/src/main/codegen/templates/TypeHelper.java
index d387b74..ad818bd 100644
--- a/exec/java-exec/src/main/codegen/templates/TypeHelper.java
+++ b/exec/java-exec/src/main/codegen/templates/TypeHelper.java
@@ -38,6 +38,11 @@ public class TypeHelper {
private static final int WIDTH_ESTIMATE = 50;
+ // Default length when casting to varchar : 65536 = 2^16
+ // This only defines an absolute maximum for values, setting
+ // a high value like this will not inflate the size for small values
+ public static final int VARCHAR_DEFAULT_CAST_LEN = 65536;
+
public static int getSize(MajorType major) {
switch (major.getMinorType()) {
<#list vv.types as type>
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java b/exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java
index d87fb76..43b8b48 100644
--- a/exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java
+++ b/exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java
@@ -31,6 +31,7 @@ import java.util.concurrent.atomic.AtomicLong;
import org.apache.drill.exec.memory.Accountor;
import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.ops.BufferManager;
import org.apache.drill.exec.ops.FragmentContext;
import org.apache.drill.exec.ops.OperatorContext;
import org.apache.drill.exec.util.AssertionUtil;
@@ -49,9 +50,15 @@ public final class DrillBuf extends AbstractByteBuf {
private volatile Accountor acct;
private volatile int length;
+ // TODO - cleanup
+ // The code is partly shared and partly copy-pasted between
+ // these three types. They should be unified under one interface
+ // to share code and to remove the hacky code here to use only
+ // one of these types at a time and use null checks to find out
+ // which.
private OperatorContext context;
private FragmentContext fContext;
-
+ private BufferManager bufManager;
public DrillBuf(BufferAllocator allocator, Accountor a, UnsafeDirectLittleEndian b) {
super(b.maxCapacity());
@@ -138,6 +145,10 @@ public final class DrillBuf extends AbstractByteBuf {
this.fContext = c;
}
+ public void setBufferManager(BufferManager bufManager) {
+ this.bufManager = bufManager;
+ }
+
public BufferAllocator getAllocator() {
return allocator;
}
@@ -150,6 +161,8 @@ public final class DrillBuf extends AbstractByteBuf {
return context.replace(this, size);
} else if(fContext != null) {
return fContext.replace(this, size);
+ } else if (bufManager != null) {
+ return bufManager.replace(this, size);
} else {
throw new UnsupportedOperationException("Realloc is only available in the context of an operator's UDFs");
}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
index 6d42136..7413a97 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
@@ -77,6 +77,7 @@ import com.google.common.base.Predicate;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
import com.google.common.collect.Lists;
+import org.apache.drill.exec.vector.VarCharVector;
public class ExpressionTreeMaterializer {
@@ -141,7 +142,7 @@ public class ExpressionTreeMaterializer {
* using an arbitrary value. We trim down the size of the stored bytes
* to the actual size so this size doesn't really matter.
*/
- castArgs.add(new ValueExpressions.LongExpression(65536, null));
+ castArgs.add(new ValueExpressions.LongExpression(TypeHelper.VARCHAR_DEFAULT_CAST_LEN, null));
}
else if (CoreDecimalUtility.isDecimalType(toType)) {
// Add the scale and precision to the arguments of the implicit cast
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
index 32cf362..8a5af56 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
@@ -28,6 +28,7 @@ import org.apache.drill.common.types.TypeProtos;
import org.apache.drill.common.util.PathScanner;
import org.apache.drill.exec.ExecConstants;
import org.apache.drill.exec.expr.DrillFunc;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
import org.apache.drill.exec.planner.sql.DrillOperatorTable;
import org.apache.drill.exec.planner.sql.DrillSqlAggOperator;
import org.apache.drill.exec.planner.sql.DrillSqlOperator;
@@ -70,6 +71,8 @@ public class DrillFunctionRegistry {
if ((existingImplementation = functionSignatureMap.get(functionSignature)) != null) {
throw new AssertionError(String.format("Conflicting functions with similar signature found. Func Name: %s, Class name: %s " +
" Class name: %s", functionName, clazz.getName(), existingImplementation));
+ } else if (holder.isAggregating() && holder.isRandom() ) {
+ logger.warn("Aggregate functions cannot be random, did not register function {}", clazz.getName());
} else {
functionSignatureMap.put(functionSignature, clazz.getName());
}
@@ -106,7 +109,15 @@ public class DrillFunctionRegistry {
if (func.isAggregating()) {
op = new DrillSqlAggOperator(name, func.getParamCount());
} else {
- op = new DrillSqlOperator(name, func.getParamCount(), func.getReturnType(), func.isDeterministic());
+ boolean isDeterministic;
+ // prevent Drill from folding constant functions with types that cannot be materialized
+ // into literals
+ if (DrillConstExecutor.NON_REDUCIBLE_TYPES.contains(func.getReturnType().getMinorType())) {
+ isDeterministic = false;
+ } else {
+ isDeterministic = func.isDeterministic();
+ }
+ op = new DrillSqlOperator(name, func.getParamCount(), func.getReturnType(), isDeterministic);
}
operatorTable.add(function.getKey(), op);
}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctionHelpers.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctionHelpers.java b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctionHelpers.java
index 3b1d7ef..2f5cf26 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctionHelpers.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctionHelpers.java
@@ -21,6 +21,7 @@ package org.apache.drill.exec.expr.fn.impl;
import io.netty.buffer.DrillBuf;
import io.netty.util.internal.PlatformDependent;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
import org.apache.drill.exec.util.AssertionUtil;
import org.joda.time.chrono.ISOChronology;
@@ -181,6 +182,20 @@ public class StringFunctionHelpers {
} // end of for_loop
}
+ /**
+ * Convert a VarCharHolder to a String.
+ *
+ * VarCharHolders are designed specifically for object reuse and mutability, only use
+ * this method when absolutely necessary for interacting with interfaces that must take
+ * a String.
+ *
+ * @param varCharHolder a mutable wrapper object that stores a variable length char array, always in UTF-8
+ * @return String of the bytes interpreted as UTF-8
+ */
+ public static String getStringFromVarCharHolder(VarCharHolder varCharHolder) {
+ return toStringFromUTF8(varCharHolder.start, varCharHolder.end, varCharHolder.buffer);
+ }
+
public static String toStringFromUTF8(int start, int end, DrillBuf buffer) {
byte[] buf = new byte[end - start];
buffer.getBytes(start, buf, 0, end - start);
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java
index 35c35ec..664b12a 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java
@@ -195,7 +195,7 @@ public class InterpreterEvaluator {
@Override
public ValueHolder visitDecimal38Constant(ValueExpressions.Decimal38Expression decExpr,Integer value) throws RuntimeException {
- return ValueHolderHelper.getDecimal28Holder(getManagedBufferIfAvailable(), decExpr.getBigDecimal().toString());
+ return ValueHolderHelper.getDecimal38Holder(getManagedBufferIfAvailable(), decExpr.getBigDecimal().toString());
}
@Override
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java b/exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java
index 536f6fd..2d22d84 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java
@@ -76,6 +76,7 @@ public class BufferManager implements AutoCloseable {
DrillBuf newBuf = allocator.buffer(size);
managedBuffers.put(newBuf.memoryAddress(), newBuf);
newBuf.setFragmentContext(fragmentContext);
+ newBuf.setBufferManager(this);
return newBuf;
}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
index 0c7a71a..4b317e0 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
@@ -67,6 +67,7 @@ import org.apache.drill.exec.util.BatchPrinter;
import org.apache.drill.exec.vector.AllocationHelper;
import org.apache.drill.exec.vector.FixedWidthVector;
import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.VarCharVector;
import org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter;
import com.carrotsearch.hppc.IntOpenHashSet;
@@ -470,7 +471,7 @@ public class ProjectRecordBatch extends AbstractSingleRecordBatch<Project> {
* using an arbitrary value. We trim down the size of the stored bytes
* to the actual size so this size doesn't really matter.
*/
- castArgs.add(new ValueExpressions.LongExpression(65536, null)); //
+ castArgs.add(new ValueExpressions.LongExpression(TypeHelper.VARCHAR_DEFAULT_CAST_LEN, null)); //
FunctionCall castCall = new FunctionCall(castFuncName, castArgs, ExpressionPosition.UNKNOWN);
exprs.add(new NamedExpression(castCall, new FieldReference(field.getPath())));
} else {
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillConstExecutor.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillConstExecutor.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillConstExecutor.java
new file mode 100644
index 0000000..0fe5a1a
--- /dev/null
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillConstExecutor.java
@@ -0,0 +1,311 @@
+/*******************************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ ******************************************************************************/
+package org.apache.drill.exec.planner.logical;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import net.hydromatic.avatica.ByteString;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.expression.ErrorCollectorImpl;
+import org.apache.drill.common.expression.ExpressionStringBuilder;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.expr.ExpressionTreeMaterializer;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.impl.DateUtility;
+import org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers;
+import org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator;
+import org.apache.drill.exec.expr.holders.BigIntHolder;
+import org.apache.drill.exec.expr.holders.BitHolder;
+import org.apache.drill.exec.expr.holders.DateHolder;
+import org.apache.drill.exec.expr.holders.Decimal18Holder;
+import org.apache.drill.exec.expr.holders.Decimal28SparseHolder;
+import org.apache.drill.exec.expr.holders.Decimal38SparseHolder;
+import org.apache.drill.exec.expr.holders.Decimal9Holder;
+import org.apache.drill.exec.expr.holders.Float4Holder;
+import org.apache.drill.exec.expr.holders.Float8Holder;
+import org.apache.drill.exec.expr.holders.IntHolder;
+import org.apache.drill.exec.expr.holders.IntervalDayHolder;
+import org.apache.drill.exec.expr.holders.IntervalYearHolder;
+import org.apache.drill.exec.expr.holders.TimeHolder;
+import org.apache.drill.exec.expr.holders.TimeStampHolder;
+import org.apache.drill.exec.expr.holders.ValueHolder;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+import org.apache.drill.exec.ops.UdfUtilities;
+import org.apache.drill.exec.vector.VarCharVector;
+import org.eigenbase.relopt.RelOptPlanner;
+import org.eigenbase.reltype.RelDataType;
+import org.eigenbase.reltype.RelDataTypeFactory;
+import org.eigenbase.rex.RexBuilder;
+import org.eigenbase.rex.RexNode;
+import org.eigenbase.sql.SqlIntervalQualifier;
+import org.eigenbase.sql.parser.SqlParserPos;
+import org.eigenbase.sql.type.SqlTypeName;
+import org.eigenbase.util.NlsString;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.List;
+
+public class DrillConstExecutor implements RelOptPlanner.Executor {
+ private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(DrillConstExecutor.class);
+
+ public static ImmutableMap<TypeProtos.MinorType, SqlTypeName> DRILL_TO_CALCITE_TYPE_MAPPING =
+ ImmutableMap.<TypeProtos.MinorType, SqlTypeName> builder()
+ .put(TypeProtos.MinorType.INT, SqlTypeName.INTEGER)
+ .put(TypeProtos.MinorType.BIGINT, SqlTypeName.BIGINT)
+ .put(TypeProtos.MinorType.FLOAT4, SqlTypeName.FLOAT)
+ .put(TypeProtos.MinorType.FLOAT8, SqlTypeName.DOUBLE)
+ .put(TypeProtos.MinorType.VARCHAR, SqlTypeName.VARCHAR)
+ .put(TypeProtos.MinorType.BIT, SqlTypeName.BOOLEAN)
+ .put(TypeProtos.MinorType.DATE, SqlTypeName.DATE)
+ .put(TypeProtos.MinorType.DECIMAL9, SqlTypeName.DECIMAL)
+ .put(TypeProtos.MinorType.DECIMAL18, SqlTypeName.DECIMAL)
+ .put(TypeProtos.MinorType.DECIMAL28SPARSE, SqlTypeName.DECIMAL)
+ .put(TypeProtos.MinorType.DECIMAL38SPARSE, SqlTypeName.DECIMAL)
+ .put(TypeProtos.MinorType.TIME, SqlTypeName.TIME)
+ .put(TypeProtos.MinorType.TIMESTAMP, SqlTypeName.TIMESTAMP)
+ .put(TypeProtos.MinorType.VARBINARY, SqlTypeName.VARBINARY)
+ .put(TypeProtos.MinorType.INTERVALYEAR, SqlTypeName.INTERVAL_YEAR_MONTH)
+ .put(TypeProtos.MinorType.INTERVALDAY, SqlTypeName.INTERVAL_DAY_TIME)
+ .put(TypeProtos.MinorType.MAP, SqlTypeName.MAP)
+ .put(TypeProtos.MinorType.LIST, SqlTypeName.ARRAY)
+ .put(TypeProtos.MinorType.LATE, SqlTypeName.ANY)
+ // These are defined in the Drill type system but have been turned off for now
+ .put(TypeProtos.MinorType.TINYINT, SqlTypeName.TINYINT)
+ .put(TypeProtos.MinorType.SMALLINT, SqlTypeName.SMALLINT)
+ // Calcite types currently not supported by Drill, nor defined in the Drill type list:
+ // - CHAR, SYMBOL, MULTISET, DISTINCT, STRUCTURED, ROW, OTHER, CURSOR, COLUMN_LIST
+ .build();
+
+ // This is a list of all types that cannot be folded at planning time for various reasons, most of the types are
+ // currently not supported at all. The reasons for the others can be found in the evaluation code in the reduce method
+ public static final List<Object> NON_REDUCIBLE_TYPES = ImmutableList.builder().add(
+ // cannot represent this as a literal according to calcite
+ TypeProtos.MinorType.INTERVAL,
+
+ // TODO - map and list are used in Drill but currently not expressible as literals, these can however be
+ // outputs of functions that take literals as inputs (such as a convert_fromJSON with a literal string
+ // as input), so we need to identify functions with these return types as non-foldable until we have a
+ // literal representation for them
+ TypeProtos.MinorType.MAP, TypeProtos.MinorType.LIST,
+
+ // TODO - DRILL-2551 - Varbinary is used in execution, but it is missing a literal definition
+ // in the logical expression representation and subsequently is not supported in
+ // RexToDrill and the logical expression visitors
+ TypeProtos.MinorType.VARBINARY,
+
+ TypeProtos.MinorType.TIMESTAMPTZ, TypeProtos.MinorType.TIMETZ, TypeProtos.MinorType.LATE,
+ TypeProtos.MinorType.TINYINT, TypeProtos.MinorType.SMALLINT, TypeProtos.MinorType.GENERIC_OBJECT, TypeProtos.MinorType.NULL,
+ TypeProtos.MinorType.DECIMAL28DENSE, TypeProtos.MinorType.DECIMAL38DENSE, TypeProtos.MinorType.MONEY,
+ TypeProtos.MinorType.FIXEDBINARY, TypeProtos.MinorType.FIXEDCHAR, TypeProtos.MinorType.FIXED16CHAR,
+ TypeProtos.MinorType.VAR16CHAR, TypeProtos.MinorType.UINT1, TypeProtos.MinorType.UINT2, TypeProtos.MinorType.UINT4,
+ TypeProtos.MinorType.UINT8)
+ .build();
+
+ FunctionImplementationRegistry funcImplReg;
+ UdfUtilities udfUtilities;
+
+ public DrillConstExecutor(FunctionImplementationRegistry funcImplReg, UdfUtilities udfUtilities) {
+ this.funcImplReg = funcImplReg;
+ this.udfUtilities = udfUtilities;
+ }
+
+ private RelDataType createCalciteTypeWithNullability(RelDataTypeFactory typeFactory,
+ SqlTypeName sqlTypeName,
+ boolean isNullable) {
+ RelDataType type;
+ if (sqlTypeName == SqlTypeName.INTERVAL_DAY_TIME) {
+ type = typeFactory.createSqlIntervalType(
+ new SqlIntervalQualifier(
+ SqlIntervalQualifier.TimeUnit.DAY,
+ SqlIntervalQualifier.TimeUnit.MINUTE,
+ SqlParserPos.ZERO));
+ } else if (sqlTypeName == SqlTypeName.INTERVAL_YEAR_MONTH) {
+ type = typeFactory.createSqlIntervalType(
+ new SqlIntervalQualifier(
+ SqlIntervalQualifier.TimeUnit.YEAR,
+ SqlIntervalQualifier.TimeUnit.MONTH,
+ SqlParserPos.ZERO));
+ } else if (sqlTypeName == SqlTypeName.VARCHAR) {
+ type = typeFactory.createSqlType(sqlTypeName, TypeHelper.VARCHAR_DEFAULT_CAST_LEN);
+ } else {
+ type = typeFactory.createSqlType(sqlTypeName);
+ }
+ return typeFactory.createTypeWithNullability(type, isNullable);
+ }
+
+ @Override
+ public void reduce(RexBuilder rexBuilder, List<RexNode> constExps, List<RexNode> reducedValues) {
+ for (RexNode newCall : constExps) {
+ LogicalExpression logEx = DrillOptiq.toDrill(new DrillParseContext(), null /* input rel */, newCall);
+
+ ErrorCollectorImpl errors = new ErrorCollectorImpl();
+ LogicalExpression materializedExpr = ExpressionTreeMaterializer.materialize(logEx, null, errors, funcImplReg);
+ if (errors.getErrorCount() != 0) {
+ String message = String.format(
+ "Failure while materializing expression in constant expression evaluator [%s]. Errors: %s",
+ newCall.toString(), errors.toString());
+ logger.error(message);
+ throw new DrillRuntimeException(message);
+ }
+
+ if (NON_REDUCIBLE_TYPES.contains(materializedExpr.getMajorType().getMinorType())) {
+ logger.debug("Constant expression not folded due to return type {}, complete expression: {}",
+ materializedExpr.getMajorType(),
+ ExpressionStringBuilder.toString(materializedExpr));
+ reducedValues.add(newCall);
+ continue;
+ }
+
+ ValueHolder output = InterpreterEvaluator.evaluateConstantExpr(udfUtilities, materializedExpr);
+ RelDataTypeFactory typeFactory = rexBuilder.getTypeFactory();
+
+ if (materializedExpr.getMajorType().getMode() == TypeProtos.DataMode.OPTIONAL && TypeHelper.isNull(output)) {
+ SqlTypeName sqlTypeName = DRILL_TO_CALCITE_TYPE_MAPPING.get(materializedExpr.getMajorType().getMinorType());
+ if (sqlTypeName == null) {
+ String message = String.format("Error reducing constant expression, unsupported type: %s.",
+ materializedExpr.getMajorType().getMinorType());
+ logger.error(message);
+ throw new DrillRuntimeException(message);
+ }
+ reducedValues.add(rexBuilder.makeNullLiteral(sqlTypeName));
+ continue;
+ }
+
+ switch(materializedExpr.getMajorType().getMinorType()) {
+ case INT:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(((IntHolder)output).value),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.INTEGER, newCall.getType().isNullable()),
+ false));
+ break;
+ case BIGINT:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(((BigIntHolder)output).value),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.BIGINT, newCall.getType().isNullable()),
+ false));
+ break;
+ case FLOAT4:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(((Float4Holder)output).value),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.FLOAT, newCall.getType().isNullable()),
+ false));
+ break;
+ case FLOAT8:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(((Float8Holder)output).value),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.DOUBLE, newCall.getType().isNullable()),
+ false));
+ break;
+ case VARCHAR:
+ reducedValues.add(rexBuilder.makeCharLiteral(
+ new NlsString(StringFunctionHelpers.getStringFromVarCharHolder((VarCharHolder)output), null, null)));
+ break;
+ case BIT:
+ reducedValues.add(rexBuilder.makeLiteral(
+ ((BitHolder)output).value == 1 ? true : false,
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.BOOLEAN, newCall.getType().isNullable()),
+ false));
+ break;
+ case DATE:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new DateTime(((DateHolder) output).value, DateTimeZone.UTC).toCalendar(null),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.DATE, newCall.getType().isNullable()),
+ false));
+ break;
+ case DECIMAL9:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(BigInteger.valueOf(((Decimal9Holder) output).value), ((Decimal9Holder)output).scale),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.DECIMAL, newCall.getType().isNullable()),
+ false));
+ break;
+ case DECIMAL18:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(BigInteger.valueOf(((Decimal18Holder) output).value), ((Decimal18Holder)output).scale),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.DECIMAL, newCall.getType().isNullable()),
+ false));
+ break;
+ case DECIMAL28SPARSE:
+ Decimal28SparseHolder decimal28Out = (Decimal28SparseHolder)output;
+ reducedValues.add(rexBuilder.makeLiteral(
+ org.apache.drill.exec.util.DecimalUtility.getBigDecimalFromSparse(
+ decimal28Out.buffer,
+ decimal28Out.start * 20,
+ 5,
+ decimal28Out.scale),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.DECIMAL, newCall.getType().isNullable()),
+ false
+ ));
+ break;
+ case DECIMAL38SPARSE:
+ Decimal38SparseHolder decimal38Out = (Decimal38SparseHolder)output;
+ reducedValues.add(rexBuilder.makeLiteral(
+ org.apache.drill.exec.util.DecimalUtility.getBigDecimalFromSparse(
+ decimal38Out.buffer,
+ decimal38Out.start * 24,
+ 6,
+ decimal38Out.scale),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.DECIMAL, newCall.getType().isNullable()),
+ false));
+ break;
+
+ case TIME:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new DateTime(((TimeHolder)output).value, DateTimeZone.UTC).toCalendar(null),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.TIME, newCall.getType().isNullable()),
+ false));
+ break;
+ case TIMESTAMP:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new DateTime(((TimeStampHolder)output).value, DateTimeZone.UTC).toCalendar(null),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.TIMESTAMP, newCall.getType().isNullable()),
+ false));
+ break;
+ case INTERVALYEAR:
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(((IntervalYearHolder)output).value),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.INTERVAL_YEAR_MONTH, newCall.getType().isNullable()),
+ false));
+ break;
+ case INTERVALDAY:
+ IntervalDayHolder intervalDayOut = (IntervalDayHolder) output;
+ reducedValues.add(rexBuilder.makeLiteral(
+ new BigDecimal(intervalDayOut.days * DateUtility.daysToStandardMillis + intervalDayOut.milliseconds),
+ createCalciteTypeWithNullability(typeFactory, SqlTypeName.INTERVAL_DAY_TIME, newCall.getType().isNullable()),
+ false));
+ break;
+ // The list of known unsupported types is used to trigger this behavior of re-using the input expression
+ // before the expression is even attempted to be evaluated, this is just here as a last precaution a
+ // as new types may be added in the future.
+ default:
+ logger.debug("Constant expression not folded due to return type {}, complete expression: {}",
+ materializedExpr.getMajorType(),
+ ExpressionStringBuilder.toString(materializedExpr));
+ reducedValues.add(newCall);
+ break;
+ }
+ }
+ }
+}
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceExpressionsRule.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceExpressionsRule.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceExpressionsRule.java
new file mode 100644
index 0000000..29175e5
--- /dev/null
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillReduceExpressionsRule.java
@@ -0,0 +1,83 @@
+/*******************************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ ******************************************************************************/
+package org.apache.drill.exec.planner.logical;
+
+import org.eigenbase.rel.CalcRel;
+import org.eigenbase.rel.FilterRel;
+import org.eigenbase.rel.RelCollationImpl;
+import org.eigenbase.rel.RelNode;
+import org.eigenbase.rel.SingleRel;
+import org.eigenbase.rel.SortRel;
+import org.eigenbase.rel.rules.ReduceExpressionsRule;
+
+import java.math.BigDecimal;
+
+public class DrillReduceExpressionsRule {
+
+ public static final DrillReduceFilterRule FILTER_INSTANCE_DRILL =
+ new DrillReduceFilterRule();
+
+ public static final DrillReduceCalcRule CALC_INSTANCE_DRILL =
+ new DrillReduceCalcRule();
+
+ private static class DrillReduceFilterRule extends ReduceExpressionsRule.ReduceFilterRule {
+
+ DrillReduceFilterRule() {
+ super("DrillReduceExpressionsRule(Filter)");
+ }
+
+ /**
+ * Drills schema flexibility requires us to override the default behavior of calcite
+ * to produce an EmptyRel in the case of a constant false filter. We need to propagate
+ * schema at runtime, so we cannot just produce a simple operator at planning time to
+ * expose the planning time known schema. Instead we have to insert a limit 0.
+ */
+ @Override
+ protected RelNode createEmptyRelOrEquivalent(FilterRel filter) {
+ return createEmptyEmptyRelHelper(filter);
+ }
+
+ }
+
+ private static class DrillReduceCalcRule extends ReduceExpressionsRule.ReduceCalcRule {
+
+ DrillReduceCalcRule() {
+ super("DrillReduceExpressionsRule(Calc)");
+ }
+
+ /**
+ * Drills schema flexibility requires us to override the default behavior of calcite
+ * to produce an EmptyRel in the case of a constant false filter. We need to propagate
+ * schema at runtime, so we cannot just produce a simple operator at planning time to
+ * expose the planning time known schema. Instead we have to insert a limit 0.
+ */
+ @Override
+ protected RelNode createEmptyRelOrEquivalent(CalcRel calc) {
+ return createEmptyEmptyRelHelper(calc);
+ }
+
+ }
+
+ private static RelNode createEmptyEmptyRelHelper(SingleRel input) {
+ return new SortRel(input.getCluster(), input.getTraitSet(),
+ input.getChild(),
+ RelCollationImpl.EMPTY,
+ input.getCluster().getRexBuilder().makeExactLiteral(BigDecimal.valueOf(0)),
+ input.getCluster().getRexBuilder().makeExactLiteral(BigDecimal.valueOf(0)));
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
index b1a7189..92a804e 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java
@@ -47,6 +47,7 @@ import org.eigenbase.rel.rules.MergeFilterRule;
import org.eigenbase.rel.rules.MergeProjectRule;
import org.eigenbase.rel.rules.PushFilterPastJoinRule;
import org.eigenbase.rel.rules.PushJoinThroughJoinRule;
+import org.eigenbase.rel.rules.ReduceExpressionsRule;
import org.eigenbase.rel.rules.RemoveDistinctAggregateRule;
import org.eigenbase.rel.rules.RemoveDistinctRule;
import org.eigenbase.rel.rules.RemoveSortRule;
@@ -62,9 +63,60 @@ public class DrillRuleSets {
public static RuleSet DRILL_BASIC_RULES = null;
+ /**
+ * Get a list of logical rules that can be turned on or off by session/system options.
+ *
+ * If a rule is intended to always be included with the logical set, it should be added
+ * to the immutable list created in the getDrillBasicRules() method below.
+ *
+ * @param queryContext - used to get the list of planner settings, other rules may
+ * also in the future need to get other query state from this,
+ * such as the available list of UDFs (as is used by the
+ * DrillMergeProjectRule created in getDrillBasicRules())
+ * @return - a list of rules that have been filtered to leave out
+ * rules that have been turned off by system or session settings
+ */
+ public static RuleSet getDrillUserConfigurableLogicalRules(QueryContext queryContext) {
+ PlannerSettings ps = queryContext.getPlannerSettings();
+
+ // This list is used to store rules that can be turned on an off
+ // by user facing planning options
+ Builder userConfigurableRules = ImmutableSet.<RelOptRule>builder();
+
+ if (ps.isConstantFoldingEnabled()) {
+ // TODO - DRILL-2218
+ userConfigurableRules.add(ReduceExpressionsRule.PROJECT_INSTANCE);
+
+ userConfigurableRules.add(DrillReduceExpressionsRule.FILTER_INSTANCE_DRILL);
+ userConfigurableRules.add(DrillReduceExpressionsRule.CALC_INSTANCE_DRILL);
+ }
+
+ return new DrillRuleSet(userConfigurableRules.build());
+ }
+
+ /**
+ * Get an immutable list of rules that will always be used when running
+ * logical planning.
+ *
+ * This would be a static member, rather than a method, but some of
+ * the rules need a reference to state that isn't available at class
+ * load time. The current example is the DrillMergeProjectRule which
+ * needs access to the registry of Drill UDFs, which is populated by
+ * scanning the class path a Drillbit startup.
+ *
+ * If a logical rule needs to be user configurable, such as turning
+ * it on and off with a system/session option, add it in the
+ * getDrillUserConfigurableLogicalRules() method instead of here.
+ *
+ * @param context - shared state used during planning, currently used here
+ * to gain access to the fucntion registry described above.
+ * @return - a RuleSet containing the logical rules that will always
+ * be used.
+ */
public static RuleSet getDrillBasicRules(QueryContext context) {
if (DRILL_BASIC_RULES == null) {
- DRILL_BASIC_RULES = new DrillRuleSet(ImmutableSet.of( //
+
+ DRILL_BASIC_RULES = new DrillRuleSet(ImmutableSet.<RelOptRule> builder().add( //
// Add support for WHERE style joins.
// PushFilterPastProjectRule.INSTANCE, // Replaced by DrillPushFilterPastProjectRule
DrillPushFilterPastProjectRule.INSTANCE,
@@ -121,9 +173,12 @@ public class DrillRuleSets {
DrillSortRule.INSTANCE,
DrillJoinRule.INSTANCE,
DrillUnionRule.INSTANCE,
+
DrillReduceAggregatesRule.INSTANCE
- ));
+ )
+ .build());
}
+
return DRILL_BASIC_RULES;
}
@@ -135,6 +190,7 @@ public class DrillRuleSets {
public static final RuleSet getPhysicalRules(QueryContext qcontext) {
List<RelOptRule> ruleList = new ArrayList<RelOptRule>();
+ PlannerSettings ps = qcontext.getPlannerSettings();
ruleList.add(ConvertCountToDirectScan.AGG_ON_PROJ_ON_SCAN);
ruleList.add(ConvertCountToDirectScan.AGG_ON_SCAN);
@@ -151,12 +207,8 @@ public class DrillRuleSets {
ruleList.add(PushLimitToTopN.INSTANCE);
ruleList.add(UnionAllPrule.INSTANCE);
-
-
// ruleList.add(UnionDistinctPrule.INSTANCE);
- PlannerSettings ps = qcontext.getPlannerSettings();
-
if (ps.isHashAggEnabled()) {
ruleList.add(HashAggPrule.INSTANCE);
}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
index f320157..8737e68 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
@@ -37,6 +37,7 @@ public class PlannerSettings implements Context{
public static final int MAX_BROADCAST_THRESHOLD = Integer.MAX_VALUE;
public static final int DEFAULT_IDENTIFIER_MAX_LENGTH = 1024;
+ public static final OptionValidator CONSTANT_FOLDING = new BooleanValidator("planner.enable_constant_folding", true);
public static final OptionValidator EXCHANGE = new BooleanValidator("planner.disable_exchanges", false);
public static final OptionValidator HASHAGG = new BooleanValidator("planner.enable_hashagg", true);
public static final OptionValidator STREAMAGG = new BooleanValidator("planner.enable_streamagg", true);
@@ -106,6 +107,10 @@ public class PlannerSettings implements Context{
return options.getOption(HASHAGG.getOptionName()).bool_val;
}
+ public boolean isConstantFoldingEnabled() {
+ return options.getOption(CONSTANT_FOLDING.getOptionName()).bool_val;
+ }
+
public boolean isStreamAggEnabled() {
return options.getOption(STREAMAGG.getOptionName()).bool_val;
}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
index d6f25fb..b98778d 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
@@ -33,6 +33,7 @@ import net.hydromatic.optiq.tools.ValidationException;
import org.apache.drill.exec.ops.QueryContext;
import org.apache.drill.exec.physical.PhysicalPlan;
import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.logical.DrillConstExecutor;
import org.apache.drill.exec.planner.logical.DrillRuleSets;
import org.apache.drill.exec.planner.physical.DrillDistributionTraitDef;
import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
@@ -90,6 +91,7 @@ public class DrillSqlWorker {
.context(context.getPlannerSettings()) //
.ruleSets(getRules(context)) //
.costFactory(costFactory) //
+ .executor(new DrillConstExecutor(context.getFunctionRegistry(), context))
.build();
this.planner = Frameworks.getPlanner(config);
HepProgramBuilder builder = new HepProgramBuilder();
@@ -102,10 +104,13 @@ public class DrillSqlWorker {
private RuleSet[] getRules(QueryContext context) {
StoragePluginRegistry storagePluginRegistry = context.getStorage();
+ RuleSet drillLogicalRules = DrillRuleSets.mergedRuleSets(
+ DrillRuleSets.getDrillBasicRules(context),
+ DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
RuleSet drillPhysicalMem = DrillRuleSets.mergedRuleSets(
DrillRuleSets.getPhysicalRules(context),
storagePluginRegistry.getStoragePluginRuleSet());
- RuleSet[] allRules = new RuleSet[] {DrillRuleSets.getDrillBasicRules(context), drillPhysicalMem};
+ RuleSet[] allRules = new RuleSet[] {drillLogicalRules, drillPhysicalMem};
return allRules;
}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java b/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
index 608fac7..1a8559e 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
@@ -41,6 +41,7 @@ public class SystemOptionManager implements OptionManager {
private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(SystemOptionManager.class);
private static final OptionValidator[] VALIDATORS = {
+ PlannerSettings.CONSTANT_FOLDING,
PlannerSettings.EXCHANGE,
PlannerSettings.HASHAGG,
PlannerSettings.STREAMAGG,
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java b/exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java
index e673230..264123f 100644
--- a/exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java
+++ b/exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java
@@ -36,6 +36,7 @@ import org.apache.drill.exec.planner.physical.PlannerSettings;
import org.apache.drill.exec.planner.sql.DrillOperatorTable;
import org.apache.drill.exec.planner.sql.DrillSqlWorker;
import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.UserBitShared;
import org.apache.drill.exec.rpc.user.UserSession;
import org.apache.drill.exec.server.DrillbitContext;
import org.apache.drill.exec.server.options.QueryOptionManager;
@@ -57,9 +58,11 @@ public class PlanningBase extends ExecTest{
@Rule public final TestRule TIMEOUT = TestTools.getTimeoutRule(10000);
@Mocked DrillbitContext dbContext;
- @Mocked QueryContext context;
private final DrillConfig config = DrillConfig.create();
+ @Mocked QueryContext context;
+
+ TopLevelAllocator allocator = new TopLevelAllocator();
protected void testSqlPlanFromFile(String file) throws Exception {
testSqlPlan(getFile(file));
@@ -82,7 +85,7 @@ public class PlanningBase extends ExecTest{
dbContext.getMetrics();
result = new MetricRegistry();
dbContext.getAllocator();
- result = new TopLevelAllocator();
+ result = allocator;
dbContext.getConfig();
result = config;
dbContext.getOptionManager();
@@ -99,7 +102,6 @@ public class PlanningBase extends ExecTest{
final SchemaPlus root = SimpleOptiqSchema.createRootSchema(false);
registry.getSchemaFactory().registerSchemas(UserSession.Builder.newBuilder().setSupportComplexTypes(true).build(), root);
-
new NonStrictExpectations() {
{
context.getNewDefaultSchema();
@@ -122,6 +124,8 @@ public class PlanningBase extends ExecTest{
result = config;
context.getDrillOperatorTable();
result = table;
+ context.getAllocator();
+ result = allocator;
}
};
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/test/java/org/apache/drill/TestProjectPushDown.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/test/java/org/apache/drill/TestProjectPushDown.java b/exec/java-exec/src/test/java/org/apache/drill/TestProjectPushDown.java
index cbb5c6e..34e57b3 100644
--- a/exec/java-exec/src/test/java/org/apache/drill/TestProjectPushDown.java
+++ b/exec/java-exec/src/test/java/org/apache/drill/TestProjectPushDown.java
@@ -197,8 +197,8 @@ public class TestProjectPushDown extends PlanTestBase {
final String pushDownSqlPattern = "select %s from cp.`%s` t0, cp.`%s` t1, cp.`%s` t2 where %s";
final String projection = "t0.fcolumns[0], t0.fmy.field, t0.freally.nested.field[0], t1.scolumns[0], t1.smy.field, t1.sreally.nested.field[0], t2.tcolumns[0], t2.tmy.field, t2.treally.nested.field[0]";
final String filter = "t0.fname = t1.sname and t1.slastname = t2.tlastname and t0.fcolumns[1] + t1.scolumns[1] = 100";
- final String firstExpected = "\"columns\" : [ \"`fname`\", \"`fcolumns`[1]\", \"`fcolumns`[0]\", \"`fmy`.`field`\", \"`freally`.`nested`.`field`[0]\" ],";
- final String secondExpected = "\"columns\" : [ \"`sname`\", \"`slastname`\", \"`scolumns`[1]\", \"`scolumns`[0]\", \"`smy`.`field`\", \"`sreally`.`nested`.`field`[0]\" ],";
+ final String firstExpected = "\"columns\" : [ \"`fname`\", \"`fcolumns`[0]\", \"`fmy`.`field`\", \"`freally`.`nested`.`field`[0]\", \"`fcolumns`[1]\" ],";
+ final String secondExpected = "\"columns\" : [ \"`sname`\", \"`slastname`\", \"`scolumns`[0]\", \"`smy`.`field`\", \"`sreally`.`nested`.`field`[0]\", \"`scolumns`[1]\" ],";
final String thirdExpected = "\"columns\" : [ \"`tlastname`\", \"`tcolumns`[0]\", \"`tmy`.`field`\", \"`treally`.`nested`.`field`[0]\" ],";
for (String table: TABLES) {
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java b/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
index d4a46c1..3b0fee0 100644
--- a/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
+++ b/exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestAggregateFunctions.java
@@ -19,6 +19,7 @@ package org.apache.drill.exec.fn.impl;
import org.apache.drill.BaseTestQuery;
import org.apache.drill.common.types.TypeProtos;
+import org.junit.Ignore;
import org.junit.Test;
public class TestAggregateFunctions extends BaseTestQuery {
@@ -58,6 +59,7 @@ public class TestAggregateFunctions extends BaseTestQuery {
.go();
}
+ @Ignore
@Test // DRILL-2092: count distinct, non distinct aggregate with group-by
public void testDrill2092() throws Exception {
String query = "select a1, b1, count(distinct c1) as dist1, \n"
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java b/exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java
new file mode 100644
index 0000000..b59be78
--- /dev/null
+++ b/exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java
@@ -0,0 +1,187 @@
+/*******************************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ ******************************************************************************/
+package org.apache.drill.exec.fn.interp;
+
+import org.apache.drill.PlanTestBase;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.PrintWriter;
+
+public class TestConstantFolding extends PlanTestBase {
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ // This should run as a @BeforeClass, but these methods must be defined static.
+ // Unfortunately, the temporary folder with an @Rule annotation cannot be static, this issue
+ // has been fixed in a newer version of JUnit
+ // http://stackoverflow.com/questions/2722358/junit-rule-temporaryfolder
+ public void createFiles(int smallFileLines, int bigFileLines) throws Exception{
+ File bigFolder = folder.newFolder("bigfile");
+ File bigFile = new File (bigFolder, "bigfile.csv");
+ PrintWriter out = new PrintWriter(bigFile);
+ for (int i = 0; i < bigFileLines; i++ ) {
+ out.println("1,2,3");
+ }
+ out.close();
+
+ File smallFolder = folder.newFolder("smallfile");
+ File smallFile = new File (smallFolder, "smallfile.csv");
+ out = new PrintWriter(smallFile);
+ for (int i = 0; i < smallFileLines; i++ ) {
+ out.println("1,2,3");
+ }
+ out.close();
+ }
+
+ @Test
+ public void testConstantFolding_allTypes() throws Exception {
+ try {
+ test("alter session set `store.json.all_text_mode` = true;");
+ String query2 = "SELECT * " +
+ "FROM cp.`/parquet/alltypes.json` " +
+ "WHERE 12 = extract(day from (to_timestamp('2014-02-12 03:18:31:07 AM', 'YYYY-MM-dd HH:mm:ss:SS a'))) " +
+ "AND cast( `int_col` AS int) = castint('1') " +
+ "AND cast( `bigint_col` AS bigint) = castbigint('100000000000') " +
+ "AND cast( `decimal9_col` AS decimal(9, 4)) = 1.0 + 0.0 " +
+ "AND cast( `decimal18_col` AS decimal(18,9)) = 123456789.000000000 + 0.0 " +
+ "AND cast( `decimal28sparse_col` AS decimal(28, 14)) = 123456789.000000000 + 0.0 " +
+ "AND cast( `decimal38sparse_col` AS decimal(38, 19)) = 123456789.000000000 + 0.0 " +
+ "AND cast( `date_col` AS date) = cast('1995-01-01' as date) " +
+ "AND cast( `date_col` AS date) = castdate('1995-01-01') " +
+ "AND cast( `date_col` AS date) = DATE '1995-01-01' " +
+ "AND cast( `timestamp_col` AS timestamp) = casttimestamp('1995-01-01 01:00:10.000') " +
+ "AND cast( `float4_col` AS float) = castfloat4('1') " +
+ "AND cast( `float8_col` AS DOUBLE) = castfloat8('1') " +
+ "AND cast( `varbinary_col` AS varbinary(65000)) = castvarbinary('qwerty', 0) " +
+ "AND cast( `intervalyear_col` AS interval year) = castintervalyear('P1Y') " +
+ "AND cast( `intervalday_col` AS interval day) = castintervalday('P1D')" +
+ "AND cast( `bit_col` AS boolean) = castbit('false') " +
+ "AND `varchar_col` = concat('qwe','rty') " +
+
+ "AND cast( `time_col` AS time) = casttime('01:00:00') " +
+
+ "";
+
+
+ testBuilder()
+ .sqlQuery(query2)
+ .ordered()
+ .baselineColumns("TINYINT_col", "SMALLINT_col", "INT_col", "FLOAT4_col",
+ "TIME_col", "DECIMAL9_col", "BIGINT_col", "UINT8_col", "FLOAT8_col",
+ "DATE_col", "TIMESTAMP_col", "DECIMAL18_col", "INTERVALYEAR_col",
+ "INTERVALDAY_col", "INTERVAL_col", "DECIMAL38SPARSE_col",
+ "DECIMAL28SPARSE_col", "VARBINARY_col", "VARCHAR_col",
+ "VAR16CHAR_col", "BIT_col")
+ // the file is being read in all_text_mode to preserve exact values (as all numbers with decimal points are read
+ // as double in the JsonReader), so the baseline values here are all specified as strings
+ .baselineValues(
+ "1", "1", "1", "1", "01:00:00", "1.0", "100000000000", "1", "1", "1995-01-01", "1995-01-01 01:00:10.000",
+ "123456789.000000000", "P1Y", "P1D", "P1Y1M1DT1H1M", "123456789.000000000",
+ "123456789.000000000", "qwerty", "qwerty","qwerty", "false"
+ )
+ .go();
+ } finally {
+ test("alter session set `store.json.all_text_mode` = false;");
+ }
+ }
+
+ @Ignore("DRILL-2553")
+ @Test
+ public void testConstExprFolding_withPartitionPrune_verySmallFiles() throws Exception {
+ createFiles(1, 8);
+ String path = folder.getRoot().toPath().toString();
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from dfs.`" + path + "/*/*.csv` where dir0 = concat('small','file')",
+ "smallfile",
+ "bigfile");
+ }
+
+ @Test
+ public void testConstExprFolding_withPartitionPrune() throws Exception {
+ createFiles(1, 1000);
+ String path = folder.getRoot().toPath().toString();
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from dfs.`" + path + "/*/*.csv` where dir0 = concat('small','file')",
+ "smallfile",
+ "bigfile");
+ }
+
+ @Test
+ public void testConstExprFolding_nonDirFilter() throws Exception {
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from cp.`functions/interp/test_input.csv` where columns[0] = 2+2",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 0\\), 4\\)",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 0\\), \\+\\(2, 2\\)\\)");
+ }
+
+ @Test
+ public void testConstantFoldingDisableOption() throws Exception {
+ try {
+ test("alter session set `planner.enable_constant_folding` = false");
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from cp.`functions/interp/test_input.csv` where columns[0] = 2+2",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 0\\), \\+\\(2, 2\\)\\)",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 0\\), 4\\)");
+ } finally {
+ test("alter session set `planner.enable_constant_folding` = true");
+ }
+ }
+
+ @Test
+ public void testConstExprFolding_moreComplicatedNonDirFilter() throws Exception {
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from cp.`functions/interp/test_input.csv` where columns[1] = ABS((6-18)/(2*3))",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 1\\), 2\\)",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 1\\), ABS\\(/\\(-\\(6, 18\\), \\*\\(2, 3\\)\\)\\)\\)");
+ }
+
+ @Test
+ public void testConstExprFolding_dontFoldRandom() throws Exception {
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from cp.`functions/interp/test_input.csv` where columns[0] = random()",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 0\\), RANDOM\\(\\)",
+ "Filter\\(condition=\\[=\\(ITEM\\(\\$[0-9]+, 0\\), [0-9\\.]+");
+ }
+
+ @Test
+ public void testConstExprFolding_ToLimit0() throws Exception {
+ testPlanOneExpectedPatternOneExcluded(
+ "select * from cp.`functions/interp/test_input.csv` where 1=0",
+ "Limit\\(offset=\\[0\\], fetch=\\[0\\]\\)",
+ "Filter\\(condition=\\[=\\(1, 0\\)\\]\\)");
+ }
+
+ // Despite a comment indicating that the plan generated by the ReduceExpressionRule
+ // should be set to be always preferable to the input rel, I cannot get it to
+ // produce a plan with the reduced result. I can trace through where the rule is fired
+ // and I can see that the expression is being evaluated and the constant is being
+ // added to a project, but this is not part of the final plan selected. May
+ // need to open a calcite bug.
+ // Tried to disable the calc and filter rules, only leave the project one, didn't help.
+ @Ignore("DRILL-2218")
+ @Test
+ public void testConstExprFolding_InSelect() throws Exception {
+ testPlanOneExcludedPattern("select columns[0], 3+5 from cp.`functions/interp/test_input.csv`",
+ "EXPR\\$[0-9]+=\\[\\+\\(3, 5\\)\\]");
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/exec/java-exec/src/test/resources/parquet/alltypes.json
----------------------------------------------------------------------
diff --git a/exec/java-exec/src/test/resources/parquet/alltypes.json b/exec/java-exec/src/test/resources/parquet/alltypes.json
new file mode 100644
index 0000000..635f197
--- /dev/null
+++ b/exec/java-exec/src/test/resources/parquet/alltypes.json
@@ -0,0 +1,8 @@
+{ "TINYINT_col":1, "SMALLINT_col":1, "INT_col":1, "FLOAT4_col":1, "TIME_col":"01:00:00", "DECIMAL9_col": "1.0", "BIGINT_col":100000000000, "UINT8_col": 1, "FLOAT8_col":1, "DATE_col":"1995-01-01", "TIMESTAMP_col":"1995-01-01 01:00:10.000", "DECIMAL18_col": "123456789.000000000", "INTERVALYEAR_col":"P1Y", "INTERVALDAY_col":"P1D", "INTERVAL_col":"P1Y1M1DT1H1M", "DECIMAL38SPARSE_col": "123456789.000000000", "DECIMAL28SPARSE_col": "123456789.000000000", "VARBINARY_col":"qwerty", "VARCHAR_col":"qwerty", "VAR16CHAR_col":"qwerty", "BIT_col":"false"}
+{ "TINYINT_col":2, "SMALLINT_col":2, "INT_col":2, "FLOAT4_col":2, "TIME_col":"02:00:00", "DECIMAL9_col": "2.0", "BIGINT_col":2, "UINT8_col": 2, "FLOAT8_col":2, "DATE_col":"1995-01-02", "TIMESTAMP_col":"1995-02-02 02:00:20.000", "DECIMAL18_col": "11.123456789", "INTERVALYEAR_col":"P2Y", "INTERVALDAY_col":"P2D", "INTERVAL_col":"P2Y2M2DT2H2M", "DECIMAL38SPARSE_col": "11.123456789", "DECIMAL28SPARSE_col": "11.123456789", "VARBINARY_col":"apache drill", "VARCHAR_col":"apache drill", "VAR16CHAR_col":"apache drill", "BIT_col":"true"}
+{ "TINYINT_col":3, "SMALLINT_col":3, "INT_col":3, "FLOAT4_col":3, "TIME_col":"03:00:00", "DECIMAL9_col": "3.0", "BIGINT_col":3, "UINT8_col": 3, "FLOAT8_col":3, "DATE_col":"1995-01-03", "TIMESTAMP_col":"1995-03-03 03:00:30.000", "DECIMAL18_col": "1", "INTERVALYEAR_col":"P3Y", "INTERVALDAY_col":"P3D", "INTERVAL_col":"P3Y3M3DT3H3M", "DECIMAL38SPARSE_col": "1", "DECIMAL28SPARSE_col": "1", "VARBINARY_col":"", "VARCHAR_col":"", "VAR16CHAR_col":"", "BIT_col":"false"}
+{ "TINYINT_col":-1, "SMALLINT_col":-1, "INT_col":-1, "FLOAT4_col":-1, "TIME_col":"10:00:00", "DECIMAL9_col": "-1.0", "BIGINT_col":-1, "UINT8_col": 10, "FLOAT8_col":10, "DATE_col":"1995-01-10", "TIMESTAMP_col":"1995-10-10 00:00:00.000", "DECIMAL18_col": "-0.111111111", "INTERVALYEAR_col":"P-1Y", "INTERVALDAY_col":"P-1D", "INTERVAL_col":"P1Y1M1DT1H1M", "DECIMAL38SPARSE_col": "-0.111111111111111", "DECIMAL28SPARSE_col": "-0.111111111111111", "VARBINARY_col":"deadbeef", "VARCHAR_col":"deadbeef", "VAR16CHAR_col":"deadbeef", "BIT_col":"false"}
+{ "TINYINT_col":-2, "SMALLINT_col":-2, "INT_col":-2, "FLOAT4_col":-2, "TIME_col":"09:00:00", "DECIMAL9_col": "-2.0", "BIGINT_col":-2, "UINT8_col": 9, "FLOAT8_col":9, "DATE_col":"1995-01-09", "TIMESTAMP_col":"1995-10-10 00:00:00.000", "DECIMAL18_col": "-987654321.123456789", "INTERVALYEAR_col":"P-2Y", "INTERVALDAY_col":"P-2D", "INTERVAL_col":"P2Y2M2DT2H2M", "DECIMAL38SPARSE_col": "-987654321.123456789", "DECIMAL28SPARSE_col": "-987654321.123456789", "VARBINARY_col":" ", "VARCHAR_col":" ", "VAR16CHAR_col":" ", "BIT_col":"true"}
+{ "TINYINT_col":-3, "SMALLINT_col":-3, "INT_col":-3, "FLOAT4_col":-3, "TIME_col":"08:00:00", "DECIMAL9_col": "-3.0", "BIGINT_col":-3, "UINT8_col": 8, "FLOAT8_col":8, "DATE_col":"1995-01-08", "TIMESTAMP_col":"1995-01-01 01:00:10.000", "DECIMAL18_col": "-222222222.222222222", "INTERVALYEAR_col":"P-3Y", "INTERVALDAY_col":"P-3D", "INTERVAL_col":"P3Y3M3DT3H3M", "DECIMAL38SPARSE_col": "-222222222222.22222222222222", "DECIMAL28SPARSE_col": "-222222222222.22222222222222", "VARBINARY_col":"hello world", "VARCHAR_col":"hello world", "VAR16CHAR_col":"hello world", "BIT_col":"false"}
+{ "TINYINT_col":10000, "SMALLINT_col":100000, "INT_col":2147483647, "FLOAT4_col":1000000000000, "TIME_col":"11:59:00", "DECIMAL9_col": "100.100", "BIGINT_col":100, "UINT8_col": 9223372036854775807, "FLOAT8_col":10000000000000, "DATE_col":"1970-01-01", "TIMESTAMP_col":"1995-02-02 02:00:20.000", "DECIMAL18_col": "777777777.666666666", "INTERVALYEAR_col":"P100Y", "INTERVALDAY_col":"P1000D", "INTERVAL_col":"P1Y1M0DT0H0M", "DECIMAL38SPARSE_col": "7777777777777.666666666666", "DECIMAL28SPARSE_col": "7777777777777.666666666666", "VARBINARY_col":"stuff", "VARCHAR_col":"stuff", "VAR16CHAR_col":"stuff", "BIT_col":"false"}
+{ "TINYINT_col":-10000, "SMALLINT_col":-100000, "INT_col":-2147483648, "FLOAT4_col":-1000000000000, "TIME_col":"23:59:00", "DECIMAL9_col": "0.0000001", "BIGINT_col":-100, "UINT8_col": 18446744073709551615 , "FLOAT8_col":-10000000000000, "DATE_col":"1900-12-31 ", "TIMESTAMP_col":"1995-03-03 03:00:30.000", "DECIMAL18_col": "121212121.343434343", "INTERVALYEAR_col":"P50Y", "INTERVALDAY_col":"P500D", "INTERVAL_col":"P1Y1M0DT0H0M", "DECIMAL38SPARSE_col": "1212121212.34343434343434", "DECIMAL28SPARSE_col": "1212121212.34343434343434", "VARBINARY_col":"json", "VARCHAR_col":"json", "VAR16CHAR_col":"json", "BIT_col":"true"}
http://git-wip-us.apache.org/repos/asf/drill/blob/8d70709d/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 35a65f2..d577769 100644
--- a/pom.xml
+++ b/pom.xml
@@ -960,7 +960,7 @@
<dependency>
<groupId>net.hydromatic</groupId>
<artifactId>optiq-core</artifactId>
- <version>0.9-drill-r20</version>
+ <version>0.9-drill-r20.2</version>
<exclusions>
<exclusion>
<groupId>org.jgrapht</groupId>