You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by gi...@apache.org on 2025/08/22 01:07:58 UTC
(druid) branch master updated: SQL: Avoid extractionFns by default. (#18092)
This is an automated email from the ASF dual-hosted git repository.
gian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 61ace5ccfc0 SQL: Avoid extractionFns by default. (#18092)
61ace5ccfc0 is described below
commit 61ace5ccfc0c7c02f47963d74c96547abb5c1909
Author: Gian Merlino <gi...@gmail.com>
AuthorDate: Thu Aug 21 18:07:52 2025 -0700
SQL: Avoid extractionFns by default. (#18092)
* SQL: Avoid extractionFns by default.
The SQL planner now avoids using extractionFns unless "sqlUseExtractionFns"
is set to "true" is set. This promotes more usage of query vectorization.
This affects the SQL functions LOOKUP, REGEXP_EXTRACT, and SUBSTRING.
* Fix BloomFilterSqlAggregatorTest.
* More idiomatic.
* Remove cannotVectorize() from a bunch of tests.
* Remove dangling comment.
---
docs/querying/sql-query-context.md | 1 +
.../bloom/sql/BloomFilterSqlAggregatorTest.java | 16 +-
.../query/expression/RegexpExtractExprMacro.java | 29 +-
.../sql/calcite/expression/SimpleExtraction.java | 14 +-
.../builtin/QueryLookupOperatorConversion.java | 2 +-
.../builtin/RegexpExtractOperatorConversion.java | 6 +-
.../builtin/SubstringOperatorConversion.java | 4 +-
.../druid/sql/calcite/planner/PlannerContext.java | 23 ++
.../druid/sql/calcite/CalciteArraysQueryTest.java | 15 +-
.../druid/sql/calcite/CalciteExplainQueryTest.java | 2 +-
.../druid/sql/calcite/CalciteJoinQueryTest.java | 44 ++-
.../calcite/CalciteLookupFunctionQueryTest.java | 312 +++++++++------------
.../calcite/CalciteMultiValueStringQueryTest.java | 49 +++-
.../apache/druid/sql/calcite/CalciteQueryTest.java | 252 +++++++++++++----
.../druid/sql/calcite/CalciteSubqueryTest.java | 15 +-
...tUsingSubqueryWithExtractionFns@all_disabled.iq | 17 +-
...stUsingSubqueryWithExtractionFns@all_enabled.iq | 17 +-
.../testUsingSubqueryWithExtractionFns@default.iq | 17 +-
...xtractionFns@filter-on-value-column_disabled.iq | 17 +-
...ryWithExtractionFns@filter-rewrites-disabled.iq | 17 +-
...ingSubqueryWithExtractionFns@filter-rewrites.iq | 17 +-
...singSubqueryWithExtractionFns@join-to-filter.iq | 17 +-
.../testGroupByLimitPushdownExtraction.iq | 17 +-
.../testRequireTimeConditionPositive3.iq | 17 +-
24 files changed, 581 insertions(+), 356 deletions(-)
diff --git a/docs/querying/sql-query-context.md b/docs/querying/sql-query-context.md
index 854d3838474..31ec12338aa 100644
--- a/docs/querying/sql-query-context.md
+++ b/docs/querying/sql-query-context.md
@@ -47,6 +47,7 @@ The table below lists the query context parameters you can use with Druid SQL.
|`useNativeQueryExplain`|If `true`, `EXPLAIN PLAN FOR` returns the explain plan as a JSON representation of equivalent native query, else it returns the original version of explain plan generated by Calcite.<br /><br />This property is provided for backwards compatibility. We don't recommend setting this parameter unless your application depends on the older behavior.|`true`|
|`sqlFinalizeOuterSketches`|If `false` (default behavior in Druid 25.0.0 and later), `DS_HLL`, `DS_THETA`, and `DS_QUANTILES_SKETCH` return sketches in query results. If `true` (default behavior in Druid 24.0.1 and earlier), Druid finalizes sketches from these functions when they appear in query results.<br /><br />This property is provided for backwards compatibility with behavior in Druid 24.0.1 and earlier. We don't recommend setting this parameter unless your application uses Druid 2 [...]
|`sqlUseBoundAndSelectors`|If `false` (default behavior in Druid 27.0.0 and later), the SQL planner uses [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). For filtering `ARRAY` typed values, `sqlUseBoundAndSelectors` must be `false`. | `false`.|
+|`sqlUseExtractionFns`|If false, the SQL planner avoids using [`extractionFn`](dimensionspecs.md#extraction-functions) in favor of using other constructs such as [virtual columns](virtual-columns.md). This parameter is provided for compatibility with prior behavior, and may be removed in a future release.|false|
|`sqlReverseLookup`|Whether to consider the [reverse-lookup rewrite](lookups.md#reverse-lookup) of the `LOOKUP` function during SQL planning.<br /><br />Druid reverses calls to `LOOKUP` only when the number of matching keys is lower than both `inSubQueryThreshold` and `sqlReverseLookupThreshold`.|`true`|
|`sqlReverseLookupThreshold`|Maximum size of `IN` filter to create when applying a [reverse-lookup rewrite](lookups.md#reverse-lookup). If a `LOOKUP` call matches more keys than the specified threshold, it remains unchanged.<br /><br />If `inSubQueryThreshold` is lower than `sqlReverseLookupThreshold`, Druid uses `inSubQueryThreshold` threshold instead.|10000|
|`sqlPullUpLookup`|Whether to consider the [pull-up rewrite](lookups.md#pull-up) of the `LOOKUP` function during SQL planning.|`true`|
diff --git a/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java b/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
index cb9e04de583..857521fbf9f 100644
--- a/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
+++ b/extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java
@@ -32,9 +32,7 @@ import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
import org.apache.druid.query.aggregation.bloom.BloomFilterAggregatorFactory;
import org.apache.druid.query.aggregation.bloom.sql.BloomFilterSqlAggregatorTest.BloomFilterSqlAggComponentSupplier;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
-import org.apache.druid.query.dimension.ExtractionDimensionSpec;
import org.apache.druid.query.expression.TestExprMacroTable;
-import org.apache.druid.query.extraction.SubstringDimExtractionFn;
import org.apache.druid.query.filter.BloomKFilter;
import org.apache.druid.query.groupby.GroupByQuery;
import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
@@ -246,15 +244,19 @@ public class BloomFilterSqlAggregatorTest extends BaseCalciteQueryTest
.dataSource(CalciteTests.DATASOURCE3)
.intervals(new MultipleIntervalSegmentSpec(ImmutableList.of(Filtration.eternity())))
.granularity(Granularities.ALL)
+ .virtualColumns(
+ new ExpressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\", 0, 1)",
+ ColumnType.STRING,
+ TestExprMacroTable.INSTANCE
+ )
+ )
.aggregators(
ImmutableList.of(
new BloomFilterAggregatorFactory(
"a0",
- new ExtractionDimensionSpec(
- "dim1",
- "a0:dim1",
- new SubstringDimExtractionFn(0, 1)
- ),
+ new DefaultDimensionSpec("v0", "a0:v0"),
TEST_NUM_ENTRIES
)
)
diff --git a/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java b/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
index e4eddc5857b..9d95ec1e9f1 100644
--- a/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
+++ b/processing/src/main/java/org/apache/druid/query/expression/RegexpExtractExprMacro.java
@@ -19,6 +19,7 @@
package org.apache.druid.query.expression;
+import org.apache.druid.error.InvalidInput;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.math.expr.Expr;
import org.apache.druid.math.expr.ExprEval;
@@ -30,6 +31,7 @@ import javax.annotation.Nullable;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
public class RegexpExtractExprMacro implements ExprMacroTable.ExprMacro
{
@@ -59,9 +61,7 @@ public class RegexpExtractExprMacro implements ExprMacroTable.ExprMacro
}
// Precompile the pattern.
- final Pattern pattern = Pattern.compile(
- StringUtils.nullToEmptyNonDruidDataString((String) patternExpr.getLiteralValue())
- );
+ final Pattern pattern = compilePattern((String) patternExpr.getLiteralValue());
final int index = indexExpr == null ? 0 : ((Number) indexExpr.getLiteralValue()).intValue();
@@ -79,7 +79,7 @@ public class RegexpExtractExprMacro implements ExprMacroTable.ExprMacro
final String s = arg.eval(bindings).asString();
if (s == null) {
- // True nulls do not match anything. Note: this branch only executes in SQL-compatible null handling mode.
+ // True nulls do not match anything.
return ExprEval.of(null);
} else {
final Matcher matcher = pattern.matcher(s);
@@ -97,4 +97,25 @@ public class RegexpExtractExprMacro implements ExprMacroTable.ExprMacro
}
return new RegexpExtractExpr(args);
}
+
+ /**
+ * Compile the provided pattern, or provide a nice error message if it cannot be compiled.
+ */
+ private static Pattern compilePattern(@Nullable String pattern)
+ {
+ try {
+ return Pattern.compile(StringUtils.nullToEmptyNonDruidDataString(pattern));
+ }
+ catch (PatternSyntaxException e) {
+ throw InvalidInput.exception(
+ e,
+ StringUtils.format(
+ "An invalid pattern [%s] was provided for the %s function, error: [%s]",
+ e.getPattern(),
+ FN_NAME,
+ e.getMessage()
+ )
+ );
+ }
+ }
}
diff --git a/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java b/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
index 81d19d1b767..c66404fd979 100644
--- a/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
+++ b/sql/src/main/java/org/apache/druid/sql/calcite/expression/SimpleExtraction.java
@@ -26,6 +26,7 @@ import org.apache.druid.query.dimension.DimensionSpec;
import org.apache.druid.query.dimension.ExtractionDimensionSpec;
import org.apache.druid.query.extraction.ExtractionFn;
import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
import javax.annotation.Nullable;
import java.util.Objects;
@@ -41,12 +42,19 @@ public class SimpleExtraction
@Nullable
private final ExtractionFn extractionFn;
- public SimpleExtraction(String column, @Nullable ExtractionFn extractionFn)
+ private SimpleExtraction(String column, @Nullable ExtractionFn extractionFn)
{
this.column = Preconditions.checkNotNull(column, "column");
this.extractionFn = extractionFn;
}
+ /**
+ * Create an instance.
+ *
+ * @param column column
+ * @param extractionFn extraction fn, if any. Callers should only provide a function if
+ * {@link PlannerContext#isUseExtractionFns()} is true.
+ */
public static SimpleExtraction of(String column, @Nullable ExtractionFn extractionFn)
{
return new SimpleExtraction(column, extractionFn);
@@ -63,6 +71,10 @@ public class SimpleExtraction
return extractionFn;
}
+ /**
+ * Create a new instance where the provided {@link ExtractionFn} is composed with the existing {@link ExtractionFn}
+ * of this instance. Callers should only use this method if {@link PlannerContext#isUseExtractionFns()} is true.
+ */
public SimpleExtraction cascade(final ExtractionFn nextExtractionFn)
{
return new SimpleExtraction(
diff --git a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
index af4ecd9426d..b9f01648c5f 100644
--- a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
+++ b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/QueryLookupOperatorConversion.java
@@ -86,7 +86,7 @@ public class QueryLookupOperatorConversion implements SqlOperatorConversion
// Add the lookup name to the set of lookups to selectively load.
plannerContext.addLookupToLoad(lookupExtractorFactoryContainerProvider.getCanonicalLookupName(lookupName));
- if (arg.isSimpleExtraction() && lookupNameExpr.isLiteral()) {
+ if (arg.isSimpleExtraction() && lookupNameExpr.isLiteral() && plannerContext.isUseExtractionFns()) {
return arg.getSimpleExtraction().cascade(
new RegisteredLookupExtractionFn(
lookupExtractorFactoryContainerProvider,
diff --git a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
index 0de60954130..cd7fcd00b13 100644
--- a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
+++ b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/RegexpExtractOperatorConversion.java
@@ -74,10 +74,12 @@ public class RegexpExtractOperatorConversion implements SqlOperatorConversion
? plannerContext.parseExpression(inputExpressions.get(2).getExpression())
: null;
- if (arg.isSimpleExtraction() && patternExpr.isLiteral() && (indexExpr == null || indexExpr.isLiteral())) {
+ if (arg.isSimpleExtraction()
+ && patternExpr.isLiteral()
+ && (indexExpr == null || indexExpr.isLiteral())
+ && plannerContext.isUseExtractionFns()) {
final String pattern = (String) patternExpr.getLiteralValue();
-
try {
return arg.getSimpleExtraction().cascade(
new RegexDimExtractionFn(
diff --git a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
index b03a74d9fbb..d4573421cb0 100644
--- a/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
+++ b/sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/SubstringOperatorConversion.java
@@ -106,7 +106,9 @@ public class SubstringOperatorConversion implements SqlOperatorConversion
return input.map(
simpleExtraction -> {
- if (adjustedIndexLiteral != null && (lengthNode == null || lengthLiteral != null)) {
+ if (adjustedIndexLiteral != null
+ && (lengthNode == null || lengthLiteral != null)
+ && plannerContext.isUseExtractionFns()) {
return simpleExtraction.cascade(new SubstringDimExtractionFn(adjustedIndexLiteral, lengthLiteral));
} else {
return null;
diff --git a/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java b/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
index 812b888ce76..859d1d5715c 100644
--- a/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
+++ b/sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerContext.java
@@ -41,6 +41,7 @@ import org.apache.druid.query.JoinAlgorithm;
import org.apache.druid.query.QueryContext;
import org.apache.druid.query.QueryContexts;
import org.apache.druid.query.explain.ExplainAttributes;
+import org.apache.druid.query.extraction.ExtractionFn;
import org.apache.druid.query.filter.InDimFilter;
import org.apache.druid.query.filter.TypedInFilter;
import org.apache.druid.query.lookup.LookupExtractor;
@@ -105,6 +106,12 @@ public class PlannerContext
public static final String CTX_SQL_USE_BOUNDS_AND_SELECTORS = "sqlUseBoundAndSelectors";
public static final boolean DEFAULT_SQL_USE_BOUNDS_AND_SELECTORS = false;
+ /**
+ * Context key for {@link PlannerContext#isUseExtractionFns()}.
+ */
+ public static final String CTX_SQL_USE_EXTRACTION_FNS = "sqlUseExtractionFns";
+ public static final boolean DEFAULT_SQL_USE_EXTRACTION_FNS = false;
+
/**
* Context key for {@link PlannerContext#isPullUpLookup()}.
*/
@@ -141,6 +148,7 @@ public class PlannerContext
private String sqlQueryId;
private boolean stringifyArrays;
private boolean useBoundsAndSelectors;
+ private boolean useExtractionFns;
private boolean pullUpLookup;
private boolean reverseLookup;
private boolean useGranularity;
@@ -356,6 +364,14 @@ public class PlannerContext
return useBoundsAndSelectors;
}
+ /**
+ * Whether we should use {@link ExtractionFn} in planning.
+ */
+ public boolean isUseExtractionFns()
+ {
+ return useExtractionFns;
+ }
+
/**
* Whether we should use {@link InDimFilter} (true) or {@link TypedInFilter} (false).
*/
@@ -689,6 +705,13 @@ public class PlannerContext
useBoundsAndSelectors = DEFAULT_SQL_USE_BOUNDS_AND_SELECTORS;
}
+ final Object useExtractionFnsParam = queryContext.get(CTX_SQL_USE_EXTRACTION_FNS);
+ if (useExtractionFnsParam != null) {
+ useExtractionFns = Numbers.parseBoolean(useExtractionFnsParam);
+ } else {
+ useExtractionFns = DEFAULT_SQL_USE_EXTRACTION_FNS;
+ }
+
final Object pullUpLookupParam = queryContext.get(CTX_SQL_PULL_UP_LOOKUP);
if (pullUpLookupParam != null) {
pullUpLookup = Numbers.parseBoolean(pullUpLookupParam);
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
index 63dc9946815..1d754142555 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
@@ -73,6 +73,7 @@ import org.apache.druid.sql.calcite.DecoupledTestConfig.QuidemTestCaseReason;
import org.apache.druid.sql.calcite.NotYetSupported.Modes;
import org.apache.druid.sql.calcite.filtration.Filtration;
import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
import org.apache.druid.sql.calcite.util.CalciteTests;
import org.apache.druid.sql.calcite.util.SqlTestFramework.StandardComponentSupplier;
import org.apache.druid.sql.http.SqlParameter;
@@ -1014,8 +1015,14 @@ public class CalciteArraysQueryTest extends BaseCalciteQueryTest
@Test
public void testArrayOverlapFilterWithExtractionFn()
{
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ Map.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
testQuery(
"SELECT dim3 FROM druid.numfoo WHERE ARRAY_OVERLAP(SUBSTRING(dim3, 1, 1), ARRAY['a','b']) LIMIT 5",
+ queryContext,
ImmutableList.of(
newScanQueryBuilder()
.dataSource(CalciteTests.DATASOURCE3)
@@ -1027,7 +1034,7 @@ public class CalciteArraysQueryTest extends BaseCalciteQueryTest
.columnTypes(ColumnType.STRING)
.resultFormat(ScanQuery.ResultFormat.RESULT_FORMAT_COMPACTED_LIST)
.limit(5)
- .context(QUERY_CONTEXT_DEFAULT)
+ .context(queryContext)
.build()
),
ImmutableList.of(
@@ -1293,6 +1300,11 @@ public class CalciteArraysQueryTest extends BaseCalciteQueryTest
@Test
public void testArrayContainsFilterWithExtractionFn()
{
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
Druids.ScanQueryBuilder builder = newScanQueryBuilder()
.dataSource(CalciteTests.DATASOURCE3)
.intervals(querySegmentSpec(Filtration.eternity()))
@@ -1310,6 +1322,7 @@ public class CalciteArraysQueryTest extends BaseCalciteQueryTest
);
testQuery(
"SELECT dim3 FROM druid.numfoo WHERE ARRAY_CONTAINS(SUBSTRING(dim3, 1, 1), ARRAY['a','b']) LIMIT 5",
+ queryContext,
ImmutableList.of(builder.build()),
ImmutableList.of(
new Object[]{"[\"a\",\"b\"]"}
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
index d57eb5d8f97..c4231fe5661 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteExplainQueryTest.java
@@ -103,7 +103,7 @@ public class CalciteExplainQueryTest extends BaseCalciteQueryTest
+ " SELECT SUBSTRING(dim1, 1, 1) FROM druid.foo WHERE dim1 IS NOT NULL\n"
+ " )\n"
+ ")";
- final String explanation = "[{\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"join\",\"left\":{\"type\":\"table\",\"name\":\"foo\"},\"right\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"table\",\"name\":\"foo\"},\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z\"]},\"filter\":{\"type\":\"not\" [...]
+ final String explanation = "[{\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"join\",\"left\":{\"type\":\"table\",\"name\":\"foo\"},\"right\":{\"type\":\"query\",\"query\":{\"queryType\":\"groupBy\",\"dataSource\":{\"type\":\"table\",\"name\":\"foo\"},\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z\"]},\"virtualColumns\":[{\"type\ [...]
final String resources = "[{\"name\":\"foo\",\"type\":\"DATASOURCE\"}]";
final String attributes = "{\"statementType\":\"SELECT\"}";
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
index eba5d515884..650135ef43d 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteJoinQueryTest.java
@@ -62,8 +62,6 @@ import org.apache.druid.query.aggregation.cardinality.CardinalityAggregatorFacto
import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
-import org.apache.druid.query.dimension.ExtractionDimensionSpec;
-import org.apache.druid.query.extraction.SubstringDimExtractionFn;
import org.apache.druid.query.filter.LikeDimFilter;
import org.apache.druid.query.groupby.GroupByQuery;
import org.apache.druid.query.groupby.ResultRow;
@@ -1535,13 +1533,14 @@ public class CalciteJoinQueryTest extends BaseCalciteQueryTest
.setDataSource(new LookupDataSource("lookyloo"))
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
- .setDimensions(
- new ExtractionDimensionSpec(
- "k",
- "d0",
- new SubstringDimExtractionFn(0, 1)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"k\", 0, 1)",
+ ColumnType.STRING
)
)
+ .setDimensions(new DefaultDimensionSpec("v0", "d0"))
.setAggregatorSpecs(new StringAnyAggregatorFactory("a0", "v", 10, true))
.build()
),
@@ -2765,19 +2764,17 @@ public class CalciteJoinQueryTest extends BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\", 0, 1)",
+ ColumnType.STRING
+ )
+ )
.setDimFilter(
not(equality("dim1", "", ColumnType.STRING))
)
- .setDimensions(
- dimensions(new ExtractionDimensionSpec(
- "dim1",
- "d0",
- new SubstringDimExtractionFn(
- 0,
- 1
- )
- ))
- )
+ .setDimensions(dimensions(new DefaultDimensionSpec("v0", "d0")))
.setContext(QUERY_CONTEXT_DEFAULT)
.build()
),
@@ -4570,16 +4567,9 @@ public class CalciteJoinQueryTest extends BaseCalciteQueryTest
)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
- .setDimensions(
- dimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- ColumnType.STRING,
- new SubstringDimExtractionFn(0, 10)
- )
- )
- )
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", "substring(\"dim1\", 0, 10)", ColumnType.STRING))
+ .setDimensions(new DefaultDimensionSpec("v0", "d0", ColumnType.STRING))
.setAggregatorSpecs(aggregators(new LongSumAggregatorFactory("a0", "cnt")))
.setContext(queryContext)
.build()
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
index 2faeae3fda4..0553164a312 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteLookupFunctionQueryTest.java
@@ -47,6 +47,7 @@ import org.apache.druid.segment.VirtualColumn;
import org.apache.druid.segment.VirtualColumns;
import org.apache.druid.segment.column.ColumnType;
import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
import org.apache.druid.sql.calcite.filtration.Filtration;
import org.apache.druid.sql.calcite.planner.PlannerConfig;
import org.apache.druid.sql.calcite.planner.PlannerContext;
@@ -74,12 +75,19 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
.put(ReverseLookupRule.CTX_MAX_OPTIMIZE_COUNT, 1)
.build();
+ /**
+ * For tests that use the lookup extraction function rather than expressions.
+ */
+ private static final Map<String, Object> QUERY_CONTEXT_WITH_EXTRACTION_FNS =
+ ImmutableMap.<String, Object>builder()
+ .putAll(QUERY_CONTEXT)
+ .put(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ .build();
+
+ private static final String LOOKUP_EXPRESSION = "lookup(\"dim1\",'lookyloo')";
private static final ExtractionFn EXTRACTION_FN =
new RegisteredLookupExtractionFn(null, "lookyloo", false, null, null, false);
- private static final ExtractionFn EXTRACTION_FN_121 =
- new RegisteredLookupExtractionFn(null, "lookyloo121", false, null, null, false);
-
@Test
public void testFilterEquals()
{
@@ -143,8 +151,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInLookupOfConcat()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(CONCAT(dim1, 'a', dim2), 'lookyloo') IN ('xa', 'xabc')"),
QUERY_CONTEXT,
@@ -167,8 +173,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterScalarInArrayLookupOfConcat()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("SCALAR_IN_ARRAY(LOOKUP(CONCAT(dim1, 'a', dim2), 'lookyloo'), ARRAY['xa', 'xabc'])"),
QUERY_CONTEXT,
@@ -205,7 +209,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInConcatOfLookup()
{
- cannotVectorize();
// One optimize call is needed for each "IN" value, because this expression is decomposed into a sequence of
// [(LOOKUP(dim1, 'lookyloo') = 'xabc' AND dim1 = 'abc') OR ...]. They can't be collected and combined.
@@ -246,8 +249,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInConcatOfLookupOfConcat()
{
- cannotVectorize();
-
// One optimize call is needed for each "IN" value, because this expression is decomposed into a sequence of
// [(LOOKUP(dim1, 'lookyloo') = 'xabc' AND dim1 = 'abc') OR ...]. They can't be collected and combined.
@@ -356,8 +357,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMultipleIsNotDistinctFrom()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS NOT DISTINCT FROM 'xabc' OR "
+ "LOOKUP(dim1, 'lookyloo') IS NOT DISTINCT FROM 'x6' OR "
@@ -371,8 +370,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterIn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent')"),
QUERY_CONTEXT,
@@ -384,8 +381,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterScalarInArray()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("SCALAR_IN_ARRAY(LOOKUP(dim1, 'lookyloo'), ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
@@ -397,7 +392,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInOverScalarInArrayThreshold()
{
- cannotVectorize();
// Set inFunctionThreshold = 1 to cause the IN to be converted to SCALAR_IN_ARRAY.
final ImmutableMap<String, Object> queryContext =
@@ -418,7 +412,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInOverMaxSize()
{
- cannotVectorize();
// Set sqlReverseLookupThreshold = 1 to stop the LOOKUP call from being reversed.
final ImmutableMap<String, Object> queryContext =
@@ -432,7 +425,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent')"),
queryContext,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
in("v0", ImmutableList.of("nonexistent", "x6", "xabc"))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -442,7 +435,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInOverMaxSize2()
{
- cannotVectorize();
// Set inSubQueryThreshold = 1 to stop the LOOKUP call from being reversed.
final ImmutableMap<String, Object> queryContext =
@@ -456,7 +448,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') = 'xabc' OR LOOKUP(dim1, 'lookyloo') = 'x6'"),
queryContext,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
in("v0", ImmutableList.of("x6", "xabc"))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -466,14 +458,12 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInOrIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent') OR LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
or(
in("dim1", Arrays.asList("6", "abc")),
isNull("v0")
@@ -486,8 +476,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInAndIsNotNull()
{
- cannotVectorize();
-
// Ideally we'd be able to eliminate "AND LOOKUP(dim1, 'lookyloo') IS NOT NULL", because it's implied by
// "LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent')". We're not currently able to do that.
@@ -496,7 +484,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
"LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent') AND LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
and(
in("dim1", ImmutableList.of("6", "abc")),
not(isNull("v0"))
@@ -509,8 +497,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInOrIsNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo121') IN ('xabc', 'x6', 'nonexistent') OR LOOKUP(dim1, 'lookyloo121') IS NULL"),
@@ -525,14 +511,12 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotInAndIsNotNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo') NOT IN ('x6', 'nonexistent') AND LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
and(
not(equality("v0", "x6", ColumnType.STRING)),
not(equality("v0", "nonexistent", ColumnType.STRING)),
@@ -546,15 +530,13 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrueAndIsNotNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"(LOOKUP(dim1, 'lookyloo') IN ('xabc', 'x6', 'nonexistent')) IS NOT TRUE "
+ "AND LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
and(
not(istrue(in("dim1", ImmutableList.of("6", "abc")))),
notNull("v0")
@@ -567,8 +549,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotInAndIsNotNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo121') NOT IN ('xabc', 'xdef', 'nonexistent') "
@@ -587,14 +567,12 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotInOrIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"LOOKUP(dim1, 'lookyloo') NOT IN ('x6', 'nonexistent') OR LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
or(
and(
not(equality("v0", "x6", ColumnType.STRING)),
@@ -613,14 +591,12 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrueOrIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"(LOOKUP(dim1, 'lookyloo') IN ('x6', 'nonexistent')) IS NOT TRUE OR LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
or(
not(istrue(equality("dim1", "6", ColumnType.STRING))),
isNull("v0")
@@ -636,13 +612,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotIn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') NOT IN ('x6', 'nonexistent')"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
and(not(equality("v0", "x6", ColumnType.STRING)), not(equality("v0", "nonexistent", ColumnType.STRING)))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -652,8 +626,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IN ('x6', 'nonexistent') IS NOT TRUE"),
QUERY_CONTEXT,
@@ -670,8 +642,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotInInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') NOT IN ('xabc', 'xdef', 'nonexistent')"),
QUERY_CONTEXT,
@@ -683,8 +653,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotInWithReplaceMissingValue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo', 'xyzzy') NOT IN ('xabc', 'x6', 'nonexistent')"),
QUERY_CONTEXT,
@@ -696,8 +664,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInIsNotTrueWithReplaceMissingValue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo', 'xyzzy') IN ('xabc', 'x6', 'nonexistent') IS NOT TRUE"),
QUERY_CONTEXT,
@@ -709,8 +675,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMvContains()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_CONTAINS(LOOKUP(dim1, 'lookyloo'), 'xabc')"),
QUERY_CONTEXT,
@@ -749,8 +713,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMvOverlap()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
@@ -768,7 +730,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc', 'x6', 'nonexistent', NULL])"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- in("dim1", Arrays.asList(null, "nonexistent", "x6", "xabc"), EXTRACTION_FN)
+ expressionFilter("mv_overlap(lookup(\"dim1\",'lookyloo'),array('xabc','x6','nonexistent',null))")
),
ImmutableList.of(
new Object[]{null, 5L},
@@ -780,8 +742,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMvOverlapNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo121'), ARRAY['xabc', 'x6', 'nonexistent', NULL])"),
QUERY_CONTEXT,
@@ -801,18 +761,18 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
buildFilterTestSql("NOT MV_CONTAINS(lookup(dim1, 'lookyloo'), 'xabc')"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
- not(equality("v0", "xabc", ColumnType.STRING))
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
+ not(expressionFilter("mv_contains(lookup(\"dim1\",'lookyloo'),'xabc')"))
),
- ImmutableList.of()
+ ImmutableList.of(
+ new Object[]{null, 5L}
+ )
);
}
@Test
public void testFilterMvContainsIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_CONTAINS(lookup(dim1, 'lookyloo'), 'xabc') IS NOT TRUE"),
QUERY_CONTEXT,
@@ -826,8 +786,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotMvContainsInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT MV_CONTAINS(LOOKUP(dim1, 'lookyloo121'), 'xabc')"),
QUERY_CONTEXT,
@@ -845,7 +803,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
buildFilterTestSql("NOT MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- not(in("dim1", ImmutableList.of("nonexistent", "x6", "xabc"), EXTRACTION_FN))
+ not(expressionFilter("mv_overlap(lookup(\"dim1\",'lookyloo'),array('xabc','x6','nonexistent'))"))
),
Collections.emptyList()
);
@@ -854,8 +812,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMvOverlapIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_OVERLAP(lookup(dim1, 'lookyloo'), ARRAY['xabc', 'x6', 'nonexistent']) IS NOT TRUE"),
QUERY_CONTEXT,
@@ -869,8 +825,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotMvOverlapInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT MV_OVERLAP(lookup(dim1, 'lookyloo121'), ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
@@ -882,7 +836,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMultipleIsDistinctFrom()
{
- cannotVectorize();
// One optimize call is needed for each "IS DISTINCT FROM", because "x IS DISTINCT FROM y" is sugar for
// "(x = y) IS NOT TRUE", and ReverseLookupRule doesn't peek into the "IS NOT TRUE" calls nested beneatth
@@ -915,13 +868,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterIsNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
isNull("v0")
),
ImmutableList.of(new Object[]{null, 5L})
@@ -944,13 +895,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterIsNotNull()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS NOT NULL"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
not(isNull("v0"))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -960,8 +909,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterIsNotNullInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') IS NOT NULL"),
QUERY_CONTEXT,
@@ -978,13 +925,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotEquals()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') <> 'x6'"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
not(equality("v0", "x6", ColumnType.STRING))
),
ImmutableList.of(new Object[]{"xabc", 1L})
@@ -994,8 +939,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotEqualsInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') <> 'xabc'"),
QUERY_CONTEXT,
@@ -1007,8 +950,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterEqualsIsNotTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') = 'x6' IS NOT TRUE"),
QUERY_CONTEXT,
@@ -1025,8 +966,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterEqualsIsNotTrueInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo121') = 'xabc' IS NOT TRUE"),
QUERY_CONTEXT,
@@ -1040,8 +979,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterIsDistinctFrom()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo') IS DISTINCT FROM 'x6'"),
QUERY_CONTEXT,
@@ -1058,11 +995,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterIsDistinctFromReplaceMissingValueWithSameLiteral()
{
- cannotVectorize();
-
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null, false);
-
testQuery(
buildFilterTestSql("LOOKUP(dim1, 'lookyloo', 'x6') IS DISTINCT FROM 'x6'"),
QUERY_CONTEXT,
@@ -1077,13 +1009,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotEquals2()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT (LOOKUP(dim1, 'lookyloo') = 'x6' OR cnt = 2)"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo')", ColumnType.STRING),
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING),
and(
not(equality("v0", "x6", ColumnType.STRING)),
not(equality("cnt", 2L, ColumnType.LONG))
@@ -1096,8 +1026,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterEqualsIsNotTrue2()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("(LOOKUP(dim1, 'lookyloo') = 'x6' OR cnt = 2) IS NOT TRUE"),
QUERY_CONTEXT,
@@ -1117,8 +1045,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterNotEquals2Injective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("NOT (LOOKUP(dim1, 'lookyloo121') = 'xdef' OR cnt = 2)"),
QUERY_CONTEXT,
@@ -1136,8 +1062,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceSameLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6') = 'x6'"),
QUERY_CONTEXT,
@@ -1165,11 +1089,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInCoalesceSameLiteral()
{
- cannotVectorize();
-
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null, false);
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6') IN ('xa', 'xabc', 'x6')"),
QUERY_CONTEXT,
@@ -1187,8 +1106,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInCoalesceSameLiteralInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo121'), 'x2') IN ('xabc', 'xdef', 'x2')"),
QUERY_CONTEXT,
@@ -1204,15 +1121,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
{
cannotVectorize();
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null, false);
-
testQuery(
buildFilterTestSql("MV_CONTAINS(COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6'), 'x6')"),
QUERY_CONTEXT,
buildFilterTestExpectedQuery(
- expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo','x6')", ColumnType.STRING),
- equality("v0", "x6", ColumnType.STRING)
+ expressionFilter("mv_contains(lookup(\"dim1\",'lookyloo','x6'),'x6')")
),
ImmutableList.of(new Object[]{null, 5L})
);
@@ -1223,25 +1136,23 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
{
cannotVectorize();
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null, false);
-
testQuery(
buildFilterTestSql("MV_OVERLAP(COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6'), ARRAY['xabc', 'x6', 'nonexistent'])"),
QUERY_CONTEXT,
- buildFilterTestExpectedQuery(in("dim1", ImmutableList.of("xabc", "x6", "nonexistent"), extractionFn)),
- ImmutableList.of(new Object[]{null, 5L}, new Object[]{"xabc", 1L})
+ buildFilterTestExpectedQuery(
+
+ expressionFilter("mv_overlap(lookup(\"dim1\",'lookyloo','x6'),array('xabc','x6','nonexistent'))")
+ ),
+ ImmutableList.of(
+ new Object[]{null, 5L},
+ new Object[]{"xabc", 1L}
+ )
);
}
@Test
public void testFilterCoalesceSameLiteralNotEquals()
{
- cannotVectorize();
-
- final RegisteredLookupExtractionFn extractionFn =
- new RegisteredLookupExtractionFn(null, "lookyloo", false, "x6", null, false);
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'x6') <> 'x6'"),
QUERY_CONTEXT,
@@ -1256,8 +1167,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceSameLiteralNotEqualsInjective()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo121'), 'xabc') <> 'xabc'"),
QUERY_CONTEXT,
@@ -1302,8 +1211,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceCastBigintDifferentLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(CAST(LOOKUP(dim1, 'lookyloo') AS BIGINT), 1) = 6"),
QUERY_CONTEXT,
@@ -1318,8 +1225,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMvContainsCoalesceDifferentLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("MV_CONTAINS(COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy'), 'x6')"),
QUERY_CONTEXT,
@@ -1331,8 +1236,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterMvOverlapCoalesceDifferentLiteral()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql(
"MV_OVERLAP(COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy'), ARRAY['xabc', 'x6', 'nonexistent'])"),
@@ -1345,8 +1248,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceDifferentLiteralNotEquals()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy') <> 'x6'"),
QUERY_CONTEXT,
@@ -1361,8 +1262,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceDifferentLiteralNotEqualsAlwaysTrue()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), 'xyzzy') <> 'nonexistent'"),
QUERY_CONTEXT,
@@ -1377,8 +1276,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceSameColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim1) = 'x6'"),
QUERY_CONTEXT,
@@ -1393,8 +1290,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterInCoalesceSameColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim1) IN ('xabc', '10.1')"),
QUERY_CONTEXT,
@@ -1412,8 +1307,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceFunctionOfSameColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim1 || '') = 'x6'"),
QUERY_CONTEXT,
@@ -1428,8 +1321,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testFilterCoalesceDifferentColumn()
{
- cannotVectorize();
-
testQuery(
buildFilterTestSql("COALESCE(LOOKUP(dim1, 'lookyloo'), dim2) = 'x6'"),
QUERY_CONTEXT,
@@ -1465,7 +1356,49 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testLookupReplaceMissingValueWith()
{
- // Cannot vectorize due to extraction dimension specs.
+ testQuery(
+ "SELECT\n"
+ + " LOOKUP(dim1, 'lookyloo', 'Missing_Value'),\n"
+ + " COALESCE(LOOKUP(dim1, 'lookyloo'), 'Missing_Value'), -- converted to the first form\n"
+ + " LOOKUP(dim1, 'lookyloo', null) as rmvNull,\n"
+ + " COUNT(*)\n"
+ + "FROM foo\n"
+ + "GROUP BY 1,2,3",
+ QUERY_CONTEXT,
+ ImmutableList.of(
+ GroupByQuery.builder()
+ .setDataSource(CalciteTests.DATASOURCE1)
+ .setInterval(querySegmentSpec(Filtration.eternity()))
+ .setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "lookup(\"dim1\",'lookyloo','Missing_Value')",
+ ColumnType.STRING
+ ),
+ expressionVirtualColumn("v1", "lookup(\"dim1\",'lookyloo',null)", ColumnType.STRING)
+ )
+ .setDimensions(
+ dimensions(
+ new DefaultDimensionSpec("v0", "d0", ColumnType.STRING),
+ new DefaultDimensionSpec("v0", "d1", ColumnType.STRING),
+ new DefaultDimensionSpec("v1", "d2", ColumnType.STRING)
+ )
+ )
+ .setAggregatorSpecs(new CountAggregatorFactory("a0"))
+ .setContext(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
+ ImmutableList.of(
+ new Object[]{"Missing_Value", "Missing_Value", null, 5L},
+ new Object[]{"xabc", "xabc", "xabc", 1L}
+ )
+ );
+ }
+
+ @Test
+ public void testLookupReplaceMissingValueWithExtractionFns()
+ {
cannotVectorize();
final RegisteredLookupExtractionFn extractionFn1 =
@@ -1478,7 +1411,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
+ " COUNT(*)\n"
+ "FROM foo\n"
+ "GROUP BY 1,2,3",
- QUERY_CONTEXT,
+ QUERY_CONTEXT_WITH_EXTRACTION_FNS,
ImmutableList.of(
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
@@ -1507,7 +1440,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
)
)
.setAggregatorSpecs(new CountAggregatorFactory("a0"))
- .setContext(QUERY_CONTEXT_DEFAULT)
+ .setContext(QUERY_CONTEXT_WITH_EXTRACTION_FNS)
.build()
),
ImmutableList.of(
@@ -1520,9 +1453,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testCountDistinctOfLookup()
{
- // Cannot vectorize due to extraction dimension spec.
- cannotVectorize();
-
testQuery(
"SELECT COUNT(DISTINCT LOOKUP(dim1, 'lookyloo')) FROM foo",
QUERY_CONTEXT,
@@ -1531,11 +1461,14 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
.dataSource(CalciteTests.DATASOURCE1)
.intervals(querySegmentSpec(Filtration.eternity()))
.granularity(Granularities.ALL)
+ .virtualColumns(
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING)
+ )
.aggregators(aggregators(
new CardinalityAggregatorFactory(
"a0",
null,
- ImmutableList.of(new ExtractionDimensionSpec("dim1", null, EXTRACTION_FN)),
+ ImmutableList.of(new DefaultDimensionSpec("v0", "v0", ColumnType.STRING)),
false,
true
)
@@ -1615,9 +1548,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
+ "SUM(foo.cnt) FROM druid.foo "
+ "GROUP BY 1";
- // ExtractionDimensionSpec cannot be vectorized
- cannotVectorize();
-
testQuery(
PLANNER_CONFIG_NO_HLL.withOverrides(
ImmutableMap.of(
@@ -1636,13 +1566,15 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ LOOKUP_EXPRESSION,
+ ColumnType.STRING
+ )
+ )
.setDimensions(dimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- ColumnType.STRING,
- EXTRACTION_FN
- ),
+ new DefaultDimensionSpec("v0", "d0", ColumnType.STRING),
new DefaultDimensionSpec("dim2", "d1", ColumnType.STRING)
))
.setAggregatorSpecs(
@@ -1650,7 +1582,7 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
new LongSumAggregatorFactory("a0", "cnt"),
new GroupingAggregatorFactory(
"a1",
- Arrays.asList("dim1", "dim2")
+ Arrays.asList("v0", "dim2")
)
)
)
@@ -1756,15 +1688,9 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
.setVirtualColumns(
- new VirtualColumn[]{
- expressionVirtualColumn(
- "v0",
- "lookup(\"dim1\",'lookyloo121')",
- ColumnType.STRING
- )
- }
+ expressionVirtualColumn("v0", "lookup(\"dim1\",'lookyloo121')", ColumnType.STRING)
)
- .setDimensions(dimensions(new ExtractionDimensionSpec("dim1", "d0", EXTRACTION_FN_121)))
+ .setDimensions(dimensions(new DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
.setAggregatorSpecs(
aggregators(
new FilteredAggregatorFactory(
@@ -1849,8 +1775,6 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Test
public void testPullUpLookupOneInjectiveOneNot()
{
- cannotVectorize();
-
testQuery(
"SELECT COUNT(*), LOOKUP(dim1, 'lookyloo'), LOOKUP(dim1, 'lookyloo121') FROM druid.foo GROUP BY 2, 3",
ImmutableList.of(
@@ -1858,8 +1782,11 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING)
+ )
.setDimensions(dimensions(
- new ExtractionDimensionSpec("dim1", "d0", EXTRACTION_FN),
+ new DefaultDimensionSpec("v0", "d0", ColumnType.STRING),
new DefaultDimensionSpec("dim1", "d1", ColumnType.STRING)
))
.setAggregatorSpecs(aggregators(new CountAggregatorFactory("a0")))
@@ -1891,16 +1818,39 @@ public class CalciteLookupFunctionQueryTest extends BaseCalciteQueryTest
@Nullable final DimFilter expectedFilter
)
{
+ final VirtualColumns virtualColumns;
+ final String groupByDimension;
+
+ if (expectedVirtualColumn != null) {
+ // The filter uses a specified virtual column. GROUP BY always uses a virtual column for LOOKUP(dim1, 'lookyloo').
+ // Determine if they are the same.
+ if (expectedVirtualColumn instanceof ExpressionVirtualColumn &&
+ LOOKUP_EXPRESSION.equals(((ExpressionVirtualColumn) expectedVirtualColumn).getExpression())) {
+ // Only need one virtual column. GROUP BY will reuse the filter's expectedVirtualColumn.
+ groupByDimension = expectedVirtualColumn.getOutputName();
+ virtualColumns = VirtualColumns.create(expectedVirtualColumn);
+ } else {
+ // Need both virtual columns: one for the filter, one for the GROUP BY.
+ groupByDimension = "v1";
+ virtualColumns = VirtualColumns.create(
+ expectedVirtualColumn,
+ expressionVirtualColumn(groupByDimension, LOOKUP_EXPRESSION, ColumnType.STRING)
+ );
+ }
+ } else {
+ // The filter does not need its own virtual column.
+ groupByDimension = "v0";
+ virtualColumns = VirtualColumns.create(expressionVirtualColumn("v0", LOOKUP_EXPRESSION, ColumnType.STRING));
+ }
+
return ImmutableList.of(
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Intervals.of("2000/3000")))
- .setVirtualColumns(expectedVirtualColumn != null
- ? VirtualColumns.create(expectedVirtualColumn)
- : VirtualColumns.EMPTY)
+ .setVirtualColumns(virtualColumns)
.setGranularity(Granularities.ALL)
.setDimFilter(expectedFilter)
- .setDimensions(new ExtractionDimensionSpec("dim1", "d0", ColumnType.STRING, EXTRACTION_FN))
+ .setDimensions(new DefaultDimensionSpec(groupByDimension, "d0", ColumnType.STRING))
.setAggregatorSpecs(new CountAggregatorFactory("a0"))
.setContext(QUERY_CONTEXT)
.build()
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
index 8b65c5731ef..ae3a6a0eec3 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteMultiValueStringQueryTest.java
@@ -20,11 +20,13 @@
package org.apache.druid.sql.calcite;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.java.util.common.granularity.Granularities;
import org.apache.druid.math.expr.ExpressionProcessing;
import org.apache.druid.query.Druids;
+import org.apache.druid.query.QueryContexts;
import org.apache.druid.query.aggregation.CountAggregatorFactory;
import org.apache.druid.query.aggregation.ExpressionLambdaAggregatorFactory;
import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
@@ -47,6 +49,7 @@ import org.apache.druid.segment.virtual.ListFilteredVirtualColumn;
import org.apache.druid.segment.virtual.PrefixFilteredVirtualColumn;
import org.apache.druid.segment.virtual.RegexFilteredVirtualColumn;
import org.apache.druid.sql.calcite.filtration.Filtration;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
import org.apache.druid.sql.calcite.util.CalciteTests;
import org.hamcrest.CoreMatchers;
import org.junit.internal.matchers.ThrowableMessageMatcher;
@@ -2117,6 +2120,43 @@ public class CalciteMultiValueStringQueryTest extends BaseCalciteQueryTest
.sql("SELECT \n"
+ " MV_FILTER_ONLY(LOOKUP(dim3,'lookyloo'),ARRAY[null]),count(1) \n"
+ "FROM druid.foo AS t group by 1\n")
+ .expectedQuery(
+ GroupByQuery.builder()
+ .setDataSource(CalciteTests.DATASOURCE1)
+ .setInterval(querySegmentSpec(Filtration.eternity()))
+ .setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "filter((x) -> array_contains(array(null), x), lookup(\"dim3\",'lookyloo'))",
+ ColumnType.STRING
+ )
+ )
+ .setDimensions(dimensions(new DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
+ .setAggregatorSpecs(aggregators(new CountAggregatorFactory("a0")))
+ .setContext(QUERY_CONTEXT_DEFAULT)
+ .build())
+ .expectedResults(
+ ImmutableList.of(new Object[]{null, 7L}))
+ .run();
+
+ }
+
+ @Test
+ public void testMultiValuedFilterOnlyWhenLookupPullsInDuplicatesWithExtractionFn()
+ {
+ cannotVectorize();
+
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
+ testBuilder()
+ .sql("SELECT \n"
+ + " MV_FILTER_ONLY(LOOKUP(dim3,'lookyloo'),ARRAY[null]),count(1) \n"
+ + "FROM druid.foo AS t group by 1\n")
+ .queryContext(queryContext)
.expectedQuery(
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
@@ -2137,17 +2177,21 @@ public class CalciteMultiValueStringQueryTest extends BaseCalciteQueryTest
)
.setDimensions(dimensions(new DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
.setAggregatorSpecs(aggregators(new CountAggregatorFactory("a0")))
- .setContext(QUERY_CONTEXT_DEFAULT)
+ .setContext(queryContext)
.build())
.expectedResults(
ImmutableList.of(new Object[]{null, 7L}))
.run();
-
}
@Test
public void testMvContainsFilterWithExtractionFn()
{
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
Druids.ScanQueryBuilder builder = newScanQueryBuilder()
.dataSource(CalciteTests.DATASOURCE3)
.intervals(querySegmentSpec(Filtration.eternity()))
@@ -2165,6 +2209,7 @@ public class CalciteMultiValueStringQueryTest extends BaseCalciteQueryTest
);
testQuery(
"SELECT dim3 FROM druid.numfoo WHERE MV_CONTAINS(SUBSTRING(dim3, 1, 1), ARRAY['a','b']) LIMIT 5",
+ queryContext,
ImmutableList.of(builder.build()),
ImmutableList.of(
new Object[]{"[\"a\",\"b\"]"}
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
index 628232d4464..f749ad0cc52 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
@@ -26,6 +26,7 @@ import com.google.common.collect.ImmutableSet;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.runtime.CalciteContextException;
import org.apache.druid.error.DruidException;
+import org.apache.druid.error.DruidExceptionMatcher;
import org.apache.druid.java.util.common.DateTimes;
import org.apache.druid.java.util.common.HumanReadableBytes;
import org.apache.druid.java.util.common.Intervals;
@@ -4172,24 +4173,25 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
public void testCountStarOnCommonTableExpression()
{
cannotVectorizeUnlessFallback();
- Druids.TimeseriesQueryBuilder builder =
- Druids.newTimeseriesQueryBuilder()
- .dataSource(CalciteTests.DATASOURCE1)
- .intervals(querySegmentSpec(Filtration.eternity()))
- .granularity(Granularities.ALL)
- .aggregators(aggregators(new CountAggregatorFactory("a0")))
- .context(QUERY_CONTEXT_DEFAULT)
- .virtualColumns(expressionVirtualColumn("v0", "substring(\"dim1\", 0, 1)", ColumnType.STRING))
- .filters(
- and(
- equality("dim2", "a", ColumnType.STRING),
- not(equality("v0", "z", ColumnType.STRING))
- )
- );
testQuery(
"WITH beep (dim1_firstchar) AS (SELECT SUBSTRING(dim1, 1, 1) FROM foo WHERE dim2 = 'a')\n"
+ "SELECT COUNT(*) FROM beep WHERE dim1_firstchar <> 'z'",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(expressionVirtualColumn("v0", "substring(\"dim1\", 0, 1)", ColumnType.STRING))
+ .filters(
+ and(
+ equality("dim2", "a", ColumnType.STRING),
+ not(equality("v0", "z", ColumnType.STRING))
+ )
+ )
+ .aggregators(aggregators(new CountAggregatorFactory("a0")))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{1L}
)
@@ -4216,7 +4218,22 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
);
testQuery(
"SELECT COUNT(*) FROM view.aview WHERE dim1_firstchar <> 'z'",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(expressionVirtualColumn("v0", "substring(\"dim1\", 0, 1)", ColumnType.STRING))
+ .filters(
+ and(
+ equality("dim2", "a", ColumnType.STRING),
+ not(equality("v0", "z", ColumnType.STRING))
+ )
+ )
+ .aggregators(aggregators(new CountAggregatorFactory("a0")))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{1L}
)
@@ -4245,7 +4262,24 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
);
testQuery(
"SELECT COUNT(*) FROM view.dview as druid WHERE druid.numfoo <> 'z'",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(
+ expressionVirtualColumn("v0", "substring(\"dim1\", 0, 1)", ColumnType.STRING)
+ )
+ .filters(
+ and(
+ equality("dim2", "a", ColumnType.STRING),
+ not(equality("v0", "z", ColumnType.STRING))
+ )
+ )
+ .aggregators(aggregators(new CountAggregatorFactory("a0")))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{1L}
)
@@ -7490,7 +7524,8 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
.intervals(querySegmentSpec(Filtration.eternity()))
.granularity(Granularities.ALL)
.virtualColumns(
- expressionVirtualColumn("v0", "concat(substring(\"dim2\", 0, 1),'x')", ColumnType.STRING)
+ expressionVirtualColumn("v0", "substring(\"dim2\", 0, 1)", ColumnType.STRING),
+ expressionVirtualColumn("v1", "concat(substring(\"dim2\", 0, 1),'x')", ColumnType.STRING)
)
.aggregators(
aggregators(
@@ -7498,7 +7533,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
new CardinalityAggregatorFactory(
"a1",
null,
- dimensions(new DefaultDimensionSpec("dim2", "dim2")),
+ dimensions(DefaultDimensionSpec.of("dim2")),
false,
true
),
@@ -7506,7 +7541,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
new CardinalityAggregatorFactory(
"a2",
null,
- dimensions(new DefaultDimensionSpec("dim2", "dim2")),
+ dimensions(DefaultDimensionSpec.of("dim2")),
false,
true
),
@@ -7515,21 +7550,14 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
new CardinalityAggregatorFactory(
"a3",
null,
- dimensions(
- new ExtractionDimensionSpec(
- "dim2",
- "dim2",
- ColumnType.STRING,
- new SubstringDimExtractionFn(0, 1)
- )
- ),
+ dimensions(DefaultDimensionSpec.of("v0")),
false,
true
),
new CardinalityAggregatorFactory(
"a4",
null,
- dimensions(new DefaultDimensionSpec("v0", "v0", ColumnType.STRING)),
+ dimensions(DefaultDimensionSpec.of("v1")),
false,
true
),
@@ -8146,6 +8174,47 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
testQuery(
"SELECT COUNT(DISTINCT SUBSTRING(dim1, 1, 1)) FROM druid.foo WHERE dim1 <> ''",
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .virtualColumns(expressionVirtualColumn("v0", "substring(\"dim1\", 0, 1)", ColumnType.STRING))
+ .filters(not(equality("dim1", "", ColumnType.STRING)))
+ .granularity(Granularities.ALL)
+ .aggregators(
+ aggregators(
+ new CardinalityAggregatorFactory(
+ "a0",
+ null,
+ dimensions(new DefaultDimensionSpec("v0", "v0", ColumnType.STRING)),
+ false,
+ true
+ )
+ )
+ )
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
+ ImmutableList.of(
+ new Object[]{4L}
+ )
+ );
+ }
+
+ @Test
+ public void testCountDistinctOfSubstringWithExtractionFn()
+ {
+ // Cannot vectorize due to extraction dimension spec.
+ cannotVectorize();
+
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
+ testQuery(
+ "SELECT COUNT(DISTINCT SUBSTRING(dim1, 1, 1)) FROM druid.foo WHERE dim1 <> ''",
+ queryContext,
ImmutableList.of(
Druids.newTimeseriesQueryBuilder()
.dataSource(CalciteTests.DATASOURCE1)
@@ -8169,7 +8238,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
)
)
)
- .context(QUERY_CONTEXT_DEFAULT)
+ .context(queryContext)
.build()
),
ImmutableList.of(
@@ -8244,10 +8313,58 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
@Test
public void testRegexpExtract()
+ {
+ // Cannot vectorize due to regexp_extract function.
+ cannotVectorize();
+
+ testQuery(
+ "SELECT DISTINCT\n"
+ + " REGEXP_EXTRACT(dim1, '^.'),\n"
+ + " REGEXP_EXTRACT(dim1, '^(.)', 1)\n"
+ + "FROM foo\n"
+ + "WHERE REGEXP_EXTRACT(dim1, '^(.)', 1) <> 'x'",
+ ImmutableList.of(
+ GroupByQuery
+ .builder()
+ .setDataSource(CalciteTests.DATASOURCE1)
+ .setInterval(querySegmentSpec(Filtration.eternity()))
+ .setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", "regexp_extract(\"dim1\",'^(.)',1)", ColumnType.STRING),
+ expressionVirtualColumn("v1", "regexp_extract(\"dim1\",'^.')", ColumnType.STRING)
+ )
+ .setDimFilter(
+ not(equality("v0", "x", ColumnType.STRING))
+ )
+ .setDimensions(
+ dimensions(
+ new DefaultDimensionSpec("v1", "d0"),
+ new DefaultDimensionSpec("v0", "d1")
+ )
+ )
+ .setContext(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
+ ImmutableList.of(
+ new Object[]{"1", "1"},
+ new Object[]{"2", "2"},
+ new Object[]{"a", "a"},
+ new Object[]{"d", "d"}
+ )
+ );
+ }
+
+ @Test
+ public void testRegexpExtractWithExtractionFn()
{
// Cannot vectorize due to extractionFn in dimension spec.
cannotVectorize();
+ final Map<String, Object> queryContext = QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ );
+
GroupByQuery.Builder builder =
GroupByQuery.builder()
.setDataSource(CalciteTests.DATASOURCE1)
@@ -8281,6 +8398,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
+ " REGEXP_EXTRACT(dim1, '^(.)', 1)\n"
+ "FROM foo\n"
+ "WHERE REGEXP_EXTRACT(dim1, '^(.)', 1) <> 'x'",
+ queryContext,
ImmutableList.of(builder.build()),
ImmutableList.of(
new Object[]{"1", "1"},
@@ -8301,6 +8419,28 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
"SELECT DISTINCT\n"
+ " REGEXP_EXTRACT(dim1, '^(.))', 1)\n"
+ "FROM foo",
+ DruidExceptionMatcher.invalidInput().expectMessageContains(
+ "An invalid pattern [^(.))] was provided for the regexp_extract function, " +
+ "error: [Unmatched closing ')' near index 3\n^(.))\n ^]"
+ )
+ );
+ }
+
+ @Test
+ public void testRegexpExtractWithBadRegexPatternWithExtractionFn()
+ {
+ // Cannot vectorize due to extractionFn in dimension spec.
+ cannotVectorize();
+
+ testQueryThrows(
+ "SELECT DISTINCT\n"
+ + " REGEXP_EXTRACT(dim1, '^(.))', 1)\n"
+ + "FROM foo",
+ QueryContexts.override(
+ QUERY_CONTEXT_DEFAULT,
+ ImmutableMap.of(PlannerContext.CTX_SQL_USE_EXTRACTION_FNS, true)
+ ),
+ DruidException.class,
invalidSqlContains(
"An invalid pattern [^(.))] was provided for the REGEXP_EXTRACT function, " +
"error: [Unmatched closing ')' near index 3\n^(.))\n ^]"
@@ -8333,7 +8473,25 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
"SELECT COUNT(*)\n"
+ "FROM foo\n"
+ "WHERE REGEXP_EXTRACT(dim1, '^1') IS NOT NULL OR REGEXP_EXTRACT('Z' || dim1, '^Z2') IS NOT NULL",
- ImmutableList.of(builder.build()),
+ ImmutableList.of(
+ Druids.newTimeseriesQueryBuilder()
+ .dataSource(CalciteTests.DATASOURCE1)
+ .intervals(querySegmentSpec(Filtration.eternity()))
+ .granularity(Granularities.ALL)
+ .virtualColumns(
+ expressionVirtualColumn("v0", "regexp_extract(\"dim1\",'^1')", ColumnType.STRING),
+ expressionVirtualColumn("v1", "regexp_extract(concat('Z',\"dim1\"),'^Z2')", ColumnType.STRING)
+ )
+ .filters(
+ or(
+ notNull("v0"),
+ notNull("v1")
+ )
+ )
+ .aggregators(new CountAggregatorFactory("a0"))
+ .context(QUERY_CONTEXT_DEFAULT)
+ .build()
+ ),
ImmutableList.of(
new Object[]{3L}
)
@@ -8474,13 +8632,16 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE3)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn("v0", "'a'", ColumnType.STRING),
+ expressionVirtualColumn("v1", "substring(\"dim5\", 0, 1)", ColumnType.STRING)
+ )
.setDimensions(
dimensions(
new DefaultDimensionSpec("v0", "d0"),
- new ExtractionDimensionSpec("dim5", "d1", new SubstringDimExtractionFn(0, 1))
+ new DefaultDimensionSpec("v1", "d1")
)
)
- .setVirtualColumns(expressionVirtualColumn("v0", "'a'", ColumnType.STRING))
.setDimFilter(equality("dim4", "a", ColumnType.STRING))
.setAggregatorSpecs(
aggregators(
@@ -9210,15 +9371,8 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
.setDataSource(new LookupDataSource("lookyloo"))
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
- .setDimensions(
- dimensions(
- new ExtractionDimensionSpec(
- "v",
- "d0",
- new SubstringDimExtractionFn(0, 1)
- )
- )
- )
+ .setVirtualColumns(expressionVirtualColumn("v0", "substring(\"v\", 0, 1)", ColumnType.STRING))
+ .setDimensions(dimensions(new DefaultDimensionSpec("v0", "d0", ColumnType.STRING)))
.setAggregatorSpecs(aggregators(new CountAggregatorFactory("a0")))
.setContext(QUERY_CONTEXT_DEFAULT)
.build()
@@ -12526,7 +12680,7 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
@Test
public void testRequireTimeConditionPositive3()
{
- // Cannot vectorize next test due to extraction dimension spec.
+ // Cannot vectorize next test due to substring function.
cannotVectorize();
// semi-join requires time condition on both left and right query
@@ -12556,14 +12710,14 @@ public class CalciteQueryTest extends BaseCalciteQueryTest
)
.setDimFilter(not(equality("dim1", "", ColumnType.STRING)))
.setGranularity(Granularities.ALL)
- .setDimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- ColumnType.STRING,
- new SubstringDimExtractionFn(0, 1)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\", 0, 1)",
+ ColumnType.STRING
)
)
+ .setDimensions(new DefaultDimensionSpec("v0", "d0", ColumnType.STRING))
.setContext(QUERY_CONTEXT_DEFAULT)
.build()
),
diff --git a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
index c13dcdab62f..05bd18f0be2 100644
--- a/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
+++ b/sql/src/test/java/org/apache/druid/sql/calcite/CalciteSubqueryTest.java
@@ -54,9 +54,7 @@ import org.apache.druid.query.aggregation.firstlast.first.StringFirstAggregatorF
import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
-import org.apache.druid.query.dimension.ExtractionDimensionSpec;
import org.apache.druid.query.expression.TestExprMacroTable;
-import org.apache.druid.query.extraction.SubstringDimExtractionFn;
import org.apache.druid.query.filter.DimFilter;
import org.apache.druid.query.filter.TypedInFilter;
import org.apache.druid.query.groupby.GroupByQuery;
@@ -253,14 +251,17 @@ public class CalciteSubqueryTest extends BaseCalciteQueryTest
.setDataSource(CalciteTests.DATASOURCE1)
.setInterval(querySegmentSpec(Filtration.eternity()))
.setGranularity(Granularities.ALL)
+ .setVirtualColumns(
+ expressionVirtualColumn(
+ "v0",
+ "substring(\"dim1\", 0, 1)",
+ ColumnType.STRING
+ )
+ )
.setDimFilter(not(equality("dim1", "", ColumnType.STRING)))
.setDimensions(
dimensions(
- new ExtractionDimensionSpec(
- "dim1",
- "d0",
- new SubstringDimExtractionFn(0, 1)
- )
+ new DefaultDimensionSpec("v0", "d0")
)
)
.setContext(QUERY_CONTEXT_DEFAULT)
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
index 742859de627..71f75680c4c 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_disabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
index 060c6cebfe6..af1cc7c9708 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@all_enabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@default.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@default.iq
index 141df36c597..fb972a91be2 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@default.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@default.iq
@@ -81,6 +81,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -94,15 +100,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
index 5fbb15e9710..4e553b80ca3 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-on-value-column_disabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites-disabled.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites-disabled.iq
index ea1ef086fa6..85cb83409b6 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites-disabled.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites-disabled.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites.iq
index 072f68d2bf8..0f664b798a6 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@filter-rewrites.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@join-to-filter.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@join-to-filter.iq
index a51c482ab8c..4e41f33603b 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@join-to-filter.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteJoinQueryTest/testUsingSubqueryWithExtractionFns@join-to-filter.iq
@@ -84,6 +84,12 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -97,15 +103,10 @@ DruidAggregate(group=[{0}], EXPR$1=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
index 665cf5ef26b..147ef51b6be 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testGroupByLimitPushdownExtraction.iq
@@ -44,6 +44,12 @@ DruidProject(dim4=[CAST('a':VARCHAR):VARCHAR], EXPR$1=[$0], EXPR$2=[$1], druid=[
"type" : "intervals",
"intervals" : [ "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim5\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "equals",
"column" : "dim4",
@@ -54,15 +60,10 @@ DruidProject(dim4=[CAST('a':VARCHAR):VARCHAR], EXPR$1=[$0], EXPR$2=[$1], druid=[
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim5",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"aggregations" : [ {
"type" : "count",
diff --git a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
index 487561db05c..9d180c80b7c 100644
--- a/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
+++ b/sql/src/test/quidem/org.apache.druid.sql.calcite.DecoupledPlanningCalciteQueryTest/testRequireTimeConditionPositive3.iq
@@ -87,6 +87,12 @@ DruidAggregate(group=[{}], EXPR$0=[COUNT()], druid=[logical])
"type" : "intervals",
"intervals" : [ "2000-01-01T00:00:00.000Z/146140482-04-24T15:36:27.903Z" ]
},
+ "virtualColumns" : [ {
+ "type" : "expression",
+ "name" : "v0",
+ "expression" : "substring(\"dim1\", 0, 1)",
+ "outputType" : "STRING"
+ } ],
"filter" : {
"type" : "not",
"field" : {
@@ -100,15 +106,10 @@ DruidAggregate(group=[{}], EXPR$0=[COUNT()], druid=[logical])
"type" : "all"
},
"dimensions" : [ {
- "type" : "extraction",
- "dimension" : "dim1",
+ "type" : "default",
+ "dimension" : "v0",
"outputName" : "d0",
- "outputType" : "STRING",
- "extractionFn" : {
- "type" : "substring",
- "index" : 0,
- "length" : 1
- }
+ "outputType" : "STRING"
} ],
"limitSpec" : {
"type" : "NoopLimitSpec"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org