You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gargvishesh (via GitHub)" <gi...@apache.org> on 2024/01/16 05:34:20 UTC
[PR] Benchmark for query planning time for IN queries (druid)
gargvishesh opened a new pull request, #15688:
URL: https://github.com/apache/druid/pull/15688
Adds a set of benchmark queries for measuring the planning time with the IN operator. Current results indicate that with the recent optimizations, the IN planning time even with `100000` expressions in the IN clause is just about `~10s`.
Results
```
Benchmark (inClauseExprCount) (inSubQueryThreshold) (rowsPerSegment) (schema) Mode Cnt Score Error Units
InBenchmark.queryEqualOrInSql 1 1000000 500000 explicit avgt 5 0.355 ± 0.010 ms/op
InBenchmark.queryEqualOrInSql 1 1000000 500000 auto avgt 5 0.350 ± 0.013 ms/op
InBenchmark.queryEqualOrInSql 10 1000000 500000 explicit avgt 5 0.731 ± 0.105 ms/op
InBenchmark.queryEqualOrInSql 10 1000000 500000 auto avgt 5 0.707 ± 0.013 ms/op
InBenchmark.queryEqualOrInSql 100 1000000 500000 explicit avgt 5 4.699 ± 1.036 ms/op
InBenchmark.queryEqualOrInSql 100 1000000 500000 auto avgt 5 5.133 ± 0.972 ms/op
InBenchmark.queryEqualOrInSql 1000 1000000 500000 explicit avgt 5 54.510 ± 2.140 ms/op
InBenchmark.queryEqualOrInSql 1000 1000000 500000 auto avgt 5 56.576 ± 0.397 ms/op
InBenchmark.queryEqualOrInSql 10000 1000000 500000 explicit avgt 5 843.706 ± 10.657 ms/op
InBenchmark.queryEqualOrInSql 10000 1000000 500000 auto avgt 5 675.669 ± 34.936 ms/op
InBenchmark.queryEqualOrInSql 100000 1000000 500000 explicit avgt 5 9548.689 ± 318.618 ms/op
InBenchmark.queryEqualOrInSql 100000 1000000 500000 auto avgt 5 10056.580 ± 955.134 ms/op
InBenchmark.queryInSql 1 1000000 500000 explicit avgt 5 0.241 ± 0.011 ms/op
InBenchmark.queryInSql 1 1000000 500000 auto avgt 5 0.238 ± 0.002 ms/op
InBenchmark.queryInSql 10 1000000 500000 explicit avgt 5 0.385 ± 0.007 ms/op
InBenchmark.queryInSql 10 1000000 500000 auto avgt 5 0.408 ± 0.003 ms/op
InBenchmark.queryInSql 100 1000000 500000 explicit avgt 5 1.805 ± 0.020 ms/op
InBenchmark.queryInSql 100 1000000 500000 auto avgt 5 1.835 ± 0.023 ms/op
InBenchmark.queryInSql 1000 1000000 500000 explicit avgt 5 17.859 ± 0.780 ms/op
InBenchmark.queryInSql 1000 1000000 500000 auto avgt 5 18.036 ± 0.513 ms/op
InBenchmark.queryInSql 10000 1000000 500000 explicit avgt 5 221.779 ± 16.690 ms/op
InBenchmark.queryInSql 10000 1000000 500000 auto avgt 5 233.813 ± 39.536 ms/op
InBenchmark.queryInSql 100000 1000000 500000 explicit avgt 5 2952.170 ± 95.481 ms/op
InBenchmark.queryInSql 100000 1000000 500000 auto avgt 5 3032.120 ± 170.323 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 merged PR #15688:
URL: https://github.com/apache/druid/pull/15688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on PR #15688:
URL: https://github.com/apache/druid/pull/15688#issuecomment-1903316530
More that one additional clause such as multiple EQUALS followed by IN, or JOIN, EQUALS followed by IN, don't seem to add much overhead post the first addition. This I think is primarily because of multiple cost analysis in [CombineAndSimplifyBounds](https://github.com/apache/druid/blob/9d4e8053a4a84e6d106afdf8b27e62d2f396ab02/sql/src/main/java/org/apache/druid/sql/calcite/filtration/CombineAndSimplifyBounds.java#L70). As a result, the following combinations end up getting range simplified:
* IN with OR
* !IN with AND
* EQUALS OR (IN WITH OR)
* !EQUALS AND (!IN WITH AND)
Also, there is an order of magnitude planning time difference between query planned with `inSubQueryThreshold` set to 0 to enforce a JOIN vs being set to the highest to retain it as IN filter.
```
Benchmark (inClauseExprCount) (inSubQueryThreshold) (rowsPerSegment) (schema) Mode Cnt Score Error Units
InPlanningBenchmark.queryJoinEqualOrInSql 10 0 500000 auto avgt 5 1.368 ± 0.183 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 10 2147483647 500000 auto avgt 5 1.006 ± 0.020 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 100 0 500000 auto avgt 5 1.929 ± 0.018 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 100 2147483647 500000 auto avgt 5 5.254 ± 0.249 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 1000 0 500000 auto avgt 5 7.852 ± 0.414 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 1000 2147483647 500000 auto avgt 5 57.873 ± 2.162 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 10000 0 500000 auto avgt 5 59.861 ± 2.709 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 10000 2147483647 500000 auto avgt 5 763.333 ± 4.939 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 100000 0 500000 auto avgt 5 805.880 ± 65.558 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 100000 2147483647 500000 auto avgt 5 9573.531 ± 971.719 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 1000000 0 500000 auto avgt 5 7489.099 ± 1517.078 ms/op
InPlanningBenchmark.queryJoinEqualOrInSql 1000000 2147483647 500000 auto avgt 5 155358.810 ± 5682.592 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 10 0 500000 auto avgt 5 1.288 ± 0.017 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 10 2147483647 500000 auto avgt 5 0.863 ± 0.011 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 100 0 500000 auto avgt 5 1.827 ± 0.029 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 100 2147483647 500000 auto avgt 5 5.418 ± 0.050 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 1000 0 500000 auto avgt 5 7.147 ± 0.235 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 1000 2147483647 500000 auto avgt 5 58.862 ± 0.472 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 10000 0 500000 auto avgt 5 57.851 ± 1.897 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 10000 2147483647 500000 auto avgt 5 756.667 ± 20.065 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 100000 0 500000 auto avgt 5 781.156 ± 47.021 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 100000 2147483647 500000 auto avgt 5 10453.902 ± 367.649 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 1000000 0 500000 auto avgt 5 7259.413 ± 584.843 ms/op
InPlanningBenchmark.queryMultiEqualOrInSql 1000000 2147483647 500000 auto avgt 5 153890.808 ± 5516.659 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on PR #15688:
URL: https://github.com/apache/druid/pull/15688#issuecomment-1900714036
Interesting! I'm curious about a couple of things:
- What's the planning time when `inSubQueryThreshold` is low enough that we go that path? It should be faster, but I'm wondering how much faster?
- What's the planning time when the query is even more complex? Let's say, some GROUP BY, JOIN, etc, plus a giant IN.
I think knowing these will help tell us what to do next.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #15688:
URL: https://github.com/apache/druid/pull/15688#discussion_r1470789791
##########
benchmarks/src/test/java/org/apache/druid/benchmark/query/InPlanningBenchmark.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark.query;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.impl.DimensionSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.math.expr.ExpressionProcessing;
+import org.apache.druid.query.DruidProcessingConfig;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.segment.AutoTypeColumnSchema;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.generator.GeneratorBasicSchemas;
+import org.apache.druid.segment.generator.GeneratorSchemaInfo;
+import org.apache.druid.segment.generator.SegmentGenerator;
+import org.apache.druid.segment.transform.TransformSpec;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.sql.calcite.planner.CalciteRulesManager;
+import org.apache.druid.sql.calcite.planner.CatalogResolver;
+import org.apache.druid.sql.calcite.planner.DruidPlanner;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.planner.PlannerResult;
+import org.apache.druid.sql.calcite.run.SqlEngine;
+import org.apache.druid.sql.calcite.schema.DruidSchemaCatalog;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Benchmark that tests various SQL queries.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+public class InPlanningBenchmark
+{
+ private static final Logger log = new Logger(SqlExpressionBenchmark.class);
+
+ static {
+ NullHandling.initializeForTests();
+ ExpressionProcessing.initializeForTests();
+ }
+
+ private static final DruidProcessingConfig PROCESSING_CONFIG = new DruidProcessingConfig()
+ {
+ @Override
+ public int intermediateComputeSizeBytes()
+ {
+ return 512 * 1024 * 1024;
+ }
+
+ @Override
+ public int getNumMergeBuffers()
+ {
+ return 3;
+ }
+
+ @Override
+ public int getNumThreads()
+ {
+ return 1;
+ }
+
+ @Override
+ public String getFormatString()
+ {
+ return "benchmarks-processing-%s";
+ }
+ };
+
+ @Param({"500000"})
+ private int rowsPerSegment;
+
+ @Param({"0", "2147483647"})
+ private int inSubQueryThreshold;
+
+ @Param({
+ "auto"
+ })
+ private String schema;
+
+ @Param({
+ "1", "10", "100", "1000", "10000", "100000", "1000000"
+ })
+ private Integer inClauseExprCount;
+ private SqlEngine engine;
+ @Nullable
+ private PlannerFactory plannerFactory;
+ private Closer closer = Closer.create();
+
+ @Setup(Level.Trial)
+ public void setup()
+ {
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("in-testbench");
+
+ final DataSegment dataSegment = DataSegment.builder()
+ .dataSource("foo")
+ .interval(schemaInfo.getDataInterval())
+ .version("1")
+ .shardSpec(new LinearShardSpec(0))
+ .size(0)
+ .build();
+
+ final PlannerConfig plannerConfig = new PlannerConfig();
+
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+ log.info(
+ "Starting benchmark setup using cacheDir[%s], rows[%,d], schema[%s].",
+ segmentGenerator.getCacheDir(),
+ rowsPerSegment,
+ schema
+ );
+ final QueryableIndex index;
+ if ("auto".equals(schema)) {
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ index = segmentGenerator.generate(
+ dataSegment,
+ schemaInfo,
+ DimensionsSpec.builder().setDimensions(columnSchemas).build(),
+ TransformSpec.NONE,
+ IndexSpec.DEFAULT,
+ Granularities.NONE,
+ rowsPerSegment
+ );
+ } else {
+ index = segmentGenerator.generate(dataSegment, schemaInfo, Granularities.NONE, rowsPerSegment);
+ }
+
+ final QueryRunnerFactoryConglomerate conglomerate = QueryStackTests.createQueryRunnerFactoryConglomerate(
+ closer,
+ PROCESSING_CONFIG
+ );
+
+ final SpecificSegmentsQuerySegmentWalker walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(
+ dataSegment,
+ index
+ );
+ closer.register(walker);
+ final ObjectMapper jsonMapper = CalciteTests.getJsonMapper();
+ final DruidSchemaCatalog rootSchema =
+ CalciteTests.createMockRootSchema(conglomerate, walker, plannerConfig, AuthTestUtils.TEST_AUTHORIZER_MAPPER);
+ engine = CalciteTests.createMockSqlEngine(walker, conglomerate);
+ plannerFactory = new PlannerFactory(
+ rootSchema,
+ CalciteTests.createOperatorTable(),
+ CalciteTests.createExprMacroTable(),
+ plannerConfig,
+ AuthTestUtils.TEST_AUTHORIZER_MAPPER,
+ jsonMapper,
+ CalciteTests.DRUID_SCHEMA_NAME,
+ new CalciteRulesManager(ImmutableSet.of()),
+ CalciteTests.createJoinableFactoryWrapper(),
+ CatalogResolver.NULL_RESOLVER,
+ new AuthConfig()
+ );
+
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(
+ engine,
+ sql,
+ ImmutableMap.of(
+ "inSubQueryThreshold",
+ inSubQueryThreshold,
+ "useCache",
+ false
+ )
+ )) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] planResult = resultSequence.toList().get(0);
+ log.info("Query plan:\n" +
+ jsonMapper.writerWithDefaultPrettyPrinter()
+ .writeValueAsString(jsonMapper.readValue((String) planResult[0], List.class))
+ );
+ }
+ catch (JsonMappingException e) {
+ throw new RuntimeException(e);
+ }
+ catch (JsonProcessingException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @TearDown(Level.Trial)
+ public void tearDown() throws Exception
+ {
+ closer.close();
+ }
+
+ @BenchmarkMode(Mode.AverageTime)
+ @OutputTimeUnit(TimeUnit.MILLISECONDS)
+ public void queryInSql(Blackhole blackhole)
+ {
+ final Map<String, Object> context = ImmutableMap.of(
+ "inSubQueryThreshold", inSubQueryThreshold, "useCache", false
+ );
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(engine, sql, context)) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in);
+ blackhole.consume(lastRow);
+ }
+ }
+
+ @Benchmark
+ @BenchmarkMode(Mode.AverageTime)
+ @OutputTimeUnit(TimeUnit.MILLISECONDS)
+ public void queryEqualOrInSql(Blackhole blackhole)
+ {
+ final Map<String, Object> context = ImmutableMap.of(
+ "inSubQueryThreshold", inSubQueryThreshold, "useCache", false
+ );
+ StringBuilder sqlBuilder = new StringBuilder().append(
+ "explain plan for select long1 from foo where string1 = '7' or long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(engine, sql, context)) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in);
+ blackhole.consume(lastRow);
+ }
+ }
+
+
+ @Benchmark
+ @BenchmarkMode(Mode.AverageTime)
+ @OutputTimeUnit(TimeUnit.MILLISECONDS)
+ public void queryMultiEqualOrInSql(Blackhole blackhole)
+ {
+ final Map<String, Object> context = ImmutableMap.of(
+ "inSubQueryThreshold", inSubQueryThreshold, "useCache", false
+ );
+ StringBuilder sqlBuilder = new StringBuilder().append(
+ "explain plan for select long1 from foo where string1 = '7' or string1 = '8' or long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(engine, sql, context)) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in);
+ blackhole.consume(lastRow);
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #15688:
URL: https://github.com/apache/druid/pull/15688#discussion_r1470791775
##########
benchmarks/src/test/java/org/apache/druid/benchmark/query/InPlanningBenchmark.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark.query;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.impl.DimensionSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.math.expr.ExpressionProcessing;
+import org.apache.druid.query.DruidProcessingConfig;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.segment.AutoTypeColumnSchema;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.generator.GeneratorBasicSchemas;
+import org.apache.druid.segment.generator.GeneratorSchemaInfo;
+import org.apache.druid.segment.generator.SegmentGenerator;
+import org.apache.druid.segment.transform.TransformSpec;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.sql.calcite.planner.CalciteRulesManager;
+import org.apache.druid.sql.calcite.planner.CatalogResolver;
+import org.apache.druid.sql.calcite.planner.DruidPlanner;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.planner.PlannerResult;
+import org.apache.druid.sql.calcite.run.SqlEngine;
+import org.apache.druid.sql.calcite.schema.DruidSchemaCatalog;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Benchmark that tests various SQL queries.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+public class InPlanningBenchmark
+{
+ private static final Logger log = new Logger(SqlExpressionBenchmark.class);
+
+ static {
+ NullHandling.initializeForTests();
+ ExpressionProcessing.initializeForTests();
+ }
+
+ private static final DruidProcessingConfig PROCESSING_CONFIG = new DruidProcessingConfig()
+ {
+ @Override
+ public int intermediateComputeSizeBytes()
+ {
+ return 512 * 1024 * 1024;
+ }
+
+ @Override
+ public int getNumMergeBuffers()
+ {
+ return 3;
+ }
+
+ @Override
+ public int getNumThreads()
+ {
+ return 1;
+ }
+
+ @Override
+ public String getFormatString()
+ {
+ return "benchmarks-processing-%s";
+ }
+ };
+
+ @Param({"500000"})
+ private int rowsPerSegment;
+
+ @Param({"0", "2147483647"})
+ private int inSubQueryThreshold;
+
+ @Param({
+ "auto"
+ })
+ private String schema;
+
+ @Param({
+ "1", "10", "100", "1000", "10000", "100000", "1000000"
+ })
+ private Integer inClauseExprCount;
+ private SqlEngine engine;
+ @Nullable
+ private PlannerFactory plannerFactory;
+ private Closer closer = Closer.create();
+
+ @Setup(Level.Trial)
+ public void setup()
+ {
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("in-testbench");
+
+ final DataSegment dataSegment = DataSegment.builder()
+ .dataSource("foo")
+ .interval(schemaInfo.getDataInterval())
+ .version("1")
+ .shardSpec(new LinearShardSpec(0))
+ .size(0)
+ .build();
+
+ final PlannerConfig plannerConfig = new PlannerConfig();
+
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+ log.info(
+ "Starting benchmark setup using cacheDir[%s], rows[%,d], schema[%s].",
+ segmentGenerator.getCacheDir(),
+ rowsPerSegment,
+ schema
+ );
+ final QueryableIndex index;
+ if ("auto".equals(schema)) {
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ index = segmentGenerator.generate(
+ dataSegment,
+ schemaInfo,
+ DimensionsSpec.builder().setDimensions(columnSchemas).build(),
+ TransformSpec.NONE,
+ IndexSpec.DEFAULT,
+ Granularities.NONE,
+ rowsPerSegment
+ );
+ } else {
+ index = segmentGenerator.generate(dataSegment, schemaInfo, Granularities.NONE, rowsPerSegment);
+ }
+
+ final QueryRunnerFactoryConglomerate conglomerate = QueryStackTests.createQueryRunnerFactoryConglomerate(
+ closer,
+ PROCESSING_CONFIG
+ );
+
+ final SpecificSegmentsQuerySegmentWalker walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(
+ dataSegment,
+ index
+ );
+ closer.register(walker);
+ final ObjectMapper jsonMapper = CalciteTests.getJsonMapper();
+ final DruidSchemaCatalog rootSchema =
+ CalciteTests.createMockRootSchema(conglomerate, walker, plannerConfig, AuthTestUtils.TEST_AUTHORIZER_MAPPER);
+ engine = CalciteTests.createMockSqlEngine(walker, conglomerate);
+ plannerFactory = new PlannerFactory(
+ rootSchema,
+ CalciteTests.createOperatorTable(),
+ CalciteTests.createExprMacroTable(),
+ plannerConfig,
+ AuthTestUtils.TEST_AUTHORIZER_MAPPER,
+ jsonMapper,
+ CalciteTests.DRUID_SCHEMA_NAME,
+ new CalciteRulesManager(ImmutableSet.of()),
+ CalciteTests.createJoinableFactoryWrapper(),
+ CatalogResolver.NULL_RESOLVER,
+ new AuthConfig()
+ );
+
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(
+ engine,
+ sql,
+ ImmutableMap.of(
+ "inSubQueryThreshold",
+ inSubQueryThreshold,
+ "useCache",
+ false
+ )
+ )) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] planResult = resultSequence.toList().get(0);
+ log.info("Query plan:\n" +
Review Comment:
Have both moved it to debug and limited logging to inClauseExprCount <= 100 -- which btw is now renamed to `inClauseLiteralsCount`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on code in PR #15688:
URL: https://github.com/apache/druid/pull/15688#discussion_r1470793089
##########
benchmarks/src/test/java/org/apache/druid/benchmark/query/InPlanningBenchmark.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark.query;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.impl.DimensionSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.math.expr.ExpressionProcessing;
+import org.apache.druid.query.DruidProcessingConfig;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.segment.AutoTypeColumnSchema;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.generator.GeneratorBasicSchemas;
+import org.apache.druid.segment.generator.GeneratorSchemaInfo;
+import org.apache.druid.segment.generator.SegmentGenerator;
+import org.apache.druid.segment.transform.TransformSpec;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.sql.calcite.planner.CalciteRulesManager;
+import org.apache.druid.sql.calcite.planner.CatalogResolver;
+import org.apache.druid.sql.calcite.planner.DruidPlanner;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.planner.PlannerResult;
+import org.apache.druid.sql.calcite.run.SqlEngine;
+import org.apache.druid.sql.calcite.schema.DruidSchemaCatalog;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Benchmark that tests various SQL queries.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+public class InPlanningBenchmark
+{
+ private static final Logger log = new Logger(SqlExpressionBenchmark.class);
+
+ static {
+ NullHandling.initializeForTests();
+ ExpressionProcessing.initializeForTests();
+ }
+
+ private static final DruidProcessingConfig PROCESSING_CONFIG = new DruidProcessingConfig()
+ {
+ @Override
+ public int intermediateComputeSizeBytes()
+ {
+ return 512 * 1024 * 1024;
+ }
+
+ @Override
+ public int getNumMergeBuffers()
+ {
+ return 3;
+ }
+
+ @Override
+ public int getNumThreads()
+ {
+ return 1;
+ }
+
+ @Override
+ public String getFormatString()
+ {
+ return "benchmarks-processing-%s";
+ }
+ };
+
+ @Param({"500000"})
+ private int rowsPerSegment;
+
+ @Param({"0", "2147483647"})
+ private int inSubQueryThreshold;
+
+ @Param({
+ "auto"
+ })
+ private String schema;
+
+ @Param({
+ "1", "10", "100", "1000", "10000", "100000", "1000000"
+ })
+ private Integer inClauseExprCount;
+ private SqlEngine engine;
+ @Nullable
+ private PlannerFactory plannerFactory;
+ private Closer closer = Closer.create();
+
+ @Setup(Level.Trial)
+ public void setup()
+ {
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("in-testbench");
+
+ final DataSegment dataSegment = DataSegment.builder()
+ .dataSource("foo")
+ .interval(schemaInfo.getDataInterval())
+ .version("1")
+ .shardSpec(new LinearShardSpec(0))
+ .size(0)
+ .build();
+
+ final PlannerConfig plannerConfig = new PlannerConfig();
+
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+ log.info(
+ "Starting benchmark setup using cacheDir[%s], rows[%,d], schema[%s].",
+ segmentGenerator.getCacheDir(),
+ rowsPerSegment,
+ schema
+ );
+ final QueryableIndex index;
+ if ("auto".equals(schema)) {
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ index = segmentGenerator.generate(
+ dataSegment,
+ schemaInfo,
+ DimensionsSpec.builder().setDimensions(columnSchemas).build(),
+ TransformSpec.NONE,
+ IndexSpec.DEFAULT,
+ Granularities.NONE,
+ rowsPerSegment
+ );
+ } else {
+ index = segmentGenerator.generate(dataSegment, schemaInfo, Granularities.NONE, rowsPerSegment);
+ }
+
+ final QueryRunnerFactoryConglomerate conglomerate = QueryStackTests.createQueryRunnerFactoryConglomerate(
+ closer,
+ PROCESSING_CONFIG
+ );
+
+ final SpecificSegmentsQuerySegmentWalker walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(
+ dataSegment,
+ index
+ );
+ closer.register(walker);
+ final ObjectMapper jsonMapper = CalciteTests.getJsonMapper();
+ final DruidSchemaCatalog rootSchema =
+ CalciteTests.createMockRootSchema(conglomerate, walker, plannerConfig, AuthTestUtils.TEST_AUTHORIZER_MAPPER);
+ engine = CalciteTests.createMockSqlEngine(walker, conglomerate);
+ plannerFactory = new PlannerFactory(
+ rootSchema,
+ CalciteTests.createOperatorTable(),
+ CalciteTests.createExprMacroTable(),
+ plannerConfig,
+ AuthTestUtils.TEST_AUTHORIZER_MAPPER,
+ jsonMapper,
+ CalciteTests.DRUID_SCHEMA_NAME,
+ new CalciteRulesManager(ImmutableSet.of()),
+ CalciteTests.createJoinableFactoryWrapper(),
+ CatalogResolver.NULL_RESOLVER,
+ new AuthConfig()
+ );
+
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
Review Comment:
Moved this to a function and reusing it everywhere. You are right: It is required here to show the query plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Benchmark for query planning time for IN queries (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15688:
URL: https://github.com/apache/druid/pull/15688#discussion_r1467321612
##########
benchmarks/src/test/java/org/apache/druid/benchmark/query/InPlanningBenchmark.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark.query;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.impl.DimensionSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.math.expr.ExpressionProcessing;
+import org.apache.druid.query.DruidProcessingConfig;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.segment.AutoTypeColumnSchema;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.generator.GeneratorBasicSchemas;
+import org.apache.druid.segment.generator.GeneratorSchemaInfo;
+import org.apache.druid.segment.generator.SegmentGenerator;
+import org.apache.druid.segment.transform.TransformSpec;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.sql.calcite.planner.CalciteRulesManager;
+import org.apache.druid.sql.calcite.planner.CatalogResolver;
+import org.apache.druid.sql.calcite.planner.DruidPlanner;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.planner.PlannerResult;
+import org.apache.druid.sql.calcite.run.SqlEngine;
+import org.apache.druid.sql.calcite.schema.DruidSchemaCatalog;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Benchmark that tests various SQL queries.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+public class InPlanningBenchmark
+{
+ private static final Logger log = new Logger(SqlExpressionBenchmark.class);
+
+ static {
+ NullHandling.initializeForTests();
+ ExpressionProcessing.initializeForTests();
+ }
+
+ private static final DruidProcessingConfig PROCESSING_CONFIG = new DruidProcessingConfig()
+ {
+ @Override
+ public int intermediateComputeSizeBytes()
+ {
+ return 512 * 1024 * 1024;
+ }
+
+ @Override
+ public int getNumMergeBuffers()
+ {
+ return 3;
+ }
+
+ @Override
+ public int getNumThreads()
+ {
+ return 1;
+ }
+
+ @Override
+ public String getFormatString()
+ {
+ return "benchmarks-processing-%s";
+ }
+ };
+
+ @Param({"500000"})
+ private int rowsPerSegment;
+
+ @Param({"0", "2147483647"})
+ private int inSubQueryThreshold;
+
+ @Param({
+ "auto"
+ })
+ private String schema;
+
+ @Param({
+ "1", "10", "100", "1000", "10000", "100000", "1000000"
+ })
+ private Integer inClauseExprCount;
+ private SqlEngine engine;
+ @Nullable
+ private PlannerFactory plannerFactory;
+ private Closer closer = Closer.create();
+
+ @Setup(Level.Trial)
+ public void setup()
+ {
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("in-testbench");
+
+ final DataSegment dataSegment = DataSegment.builder()
+ .dataSource("foo")
+ .interval(schemaInfo.getDataInterval())
+ .version("1")
+ .shardSpec(new LinearShardSpec(0))
+ .size(0)
+ .build();
+
+ final PlannerConfig plannerConfig = new PlannerConfig();
+
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+ log.info(
+ "Starting benchmark setup using cacheDir[%s], rows[%,d], schema[%s].",
+ segmentGenerator.getCacheDir(),
+ rowsPerSegment,
+ schema
+ );
+ final QueryableIndex index;
+ if ("auto".equals(schema)) {
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ index = segmentGenerator.generate(
+ dataSegment,
+ schemaInfo,
+ DimensionsSpec.builder().setDimensions(columnSchemas).build(),
+ TransformSpec.NONE,
+ IndexSpec.DEFAULT,
+ Granularities.NONE,
+ rowsPerSegment
+ );
+ } else {
+ index = segmentGenerator.generate(dataSegment, schemaInfo, Granularities.NONE, rowsPerSegment);
+ }
+
+ final QueryRunnerFactoryConglomerate conglomerate = QueryStackTests.createQueryRunnerFactoryConglomerate(
+ closer,
+ PROCESSING_CONFIG
+ );
+
+ final SpecificSegmentsQuerySegmentWalker walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(
+ dataSegment,
+ index
+ );
+ closer.register(walker);
+ final ObjectMapper jsonMapper = CalciteTests.getJsonMapper();
+ final DruidSchemaCatalog rootSchema =
+ CalciteTests.createMockRootSchema(conglomerate, walker, plannerConfig, AuthTestUtils.TEST_AUTHORIZER_MAPPER);
+ engine = CalciteTests.createMockSqlEngine(walker, conglomerate);
+ plannerFactory = new PlannerFactory(
+ rootSchema,
+ CalciteTests.createOperatorTable(),
+ CalciteTests.createExprMacroTable(),
+ plannerConfig,
+ AuthTestUtils.TEST_AUTHORIZER_MAPPER,
+ jsonMapper,
+ CalciteTests.DRUID_SCHEMA_NAME,
+ new CalciteRulesManager(ImmutableSet.of()),
+ CalciteTests.createJoinableFactoryWrapper(),
+ CatalogResolver.NULL_RESOLVER,
+ new AuthConfig()
+ );
+
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(
+ engine,
+ sql,
+ ImmutableMap.of(
+ "inSubQueryThreshold",
+ inSubQueryThreshold,
+ "useCache",
+ false
+ )
+ )) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] planResult = resultSequence.toList().get(0);
+ log.info("Query plan:\n" +
Review Comment:
nit: I've run this benchmark a few times; and this log was not-at-all helpfull; can we push it to `debug` or omit it if `inClauseExprCount > 100` or something?
##########
benchmarks/src/test/java/org/apache/druid/benchmark/query/InPlanningBenchmark.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark.query;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.impl.DimensionSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.math.expr.ExpressionProcessing;
+import org.apache.druid.query.DruidProcessingConfig;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.segment.AutoTypeColumnSchema;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.generator.GeneratorBasicSchemas;
+import org.apache.druid.segment.generator.GeneratorSchemaInfo;
+import org.apache.druid.segment.generator.SegmentGenerator;
+import org.apache.druid.segment.transform.TransformSpec;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.sql.calcite.planner.CalciteRulesManager;
+import org.apache.druid.sql.calcite.planner.CatalogResolver;
+import org.apache.druid.sql.calcite.planner.DruidPlanner;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.planner.PlannerResult;
+import org.apache.druid.sql.calcite.run.SqlEngine;
+import org.apache.druid.sql.calcite.schema.DruidSchemaCatalog;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Benchmark that tests various SQL queries.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+public class InPlanningBenchmark
+{
+ private static final Logger log = new Logger(SqlExpressionBenchmark.class);
+
+ static {
+ NullHandling.initializeForTests();
+ ExpressionProcessing.initializeForTests();
+ }
+
+ private static final DruidProcessingConfig PROCESSING_CONFIG = new DruidProcessingConfig()
+ {
+ @Override
+ public int intermediateComputeSizeBytes()
+ {
+ return 512 * 1024 * 1024;
+ }
+
+ @Override
+ public int getNumMergeBuffers()
+ {
+ return 3;
+ }
+
+ @Override
+ public int getNumThreads()
+ {
+ return 1;
+ }
+
+ @Override
+ public String getFormatString()
+ {
+ return "benchmarks-processing-%s";
+ }
+ };
+
+ @Param({"500000"})
+ private int rowsPerSegment;
+
+ @Param({"0", "2147483647"})
+ private int inSubQueryThreshold;
+
+ @Param({
+ "auto"
+ })
+ private String schema;
+
+ @Param({
+ "1", "10", "100", "1000", "10000", "100000", "1000000"
+ })
+ private Integer inClauseExprCount;
+ private SqlEngine engine;
+ @Nullable
+ private PlannerFactory plannerFactory;
+ private Closer closer = Closer.create();
+
+ @Setup(Level.Trial)
+ public void setup()
+ {
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("in-testbench");
+
+ final DataSegment dataSegment = DataSegment.builder()
+ .dataSource("foo")
+ .interval(schemaInfo.getDataInterval())
+ .version("1")
+ .shardSpec(new LinearShardSpec(0))
+ .size(0)
+ .build();
+
+ final PlannerConfig plannerConfig = new PlannerConfig();
+
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+ log.info(
+ "Starting benchmark setup using cacheDir[%s], rows[%,d], schema[%s].",
+ segmentGenerator.getCacheDir(),
+ rowsPerSegment,
+ schema
+ );
+ final QueryableIndex index;
+ if ("auto".equals(schema)) {
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ index = segmentGenerator.generate(
+ dataSegment,
+ schemaInfo,
+ DimensionsSpec.builder().setDimensions(columnSchemas).build(),
+ TransformSpec.NONE,
+ IndexSpec.DEFAULT,
+ Granularities.NONE,
+ rowsPerSegment
+ );
+ } else {
+ index = segmentGenerator.generate(dataSegment, schemaInfo, Granularities.NONE, rowsPerSegment);
+ }
+
+ final QueryRunnerFactoryConglomerate conglomerate = QueryStackTests.createQueryRunnerFactoryConglomerate(
+ closer,
+ PROCESSING_CONFIG
+ );
+
+ final SpecificSegmentsQuerySegmentWalker walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(
+ dataSegment,
+ index
+ );
+ closer.register(walker);
+ final ObjectMapper jsonMapper = CalciteTests.getJsonMapper();
+ final DruidSchemaCatalog rootSchema =
+ CalciteTests.createMockRootSchema(conglomerate, walker, plannerConfig, AuthTestUtils.TEST_AUTHORIZER_MAPPER);
+ engine = CalciteTests.createMockSqlEngine(walker, conglomerate);
+ plannerFactory = new PlannerFactory(
+ rootSchema,
+ CalciteTests.createOperatorTable(),
+ CalciteTests.createExprMacroTable(),
+ plannerConfig,
+ AuthTestUtils.TEST_AUTHORIZER_MAPPER,
+ jsonMapper,
+ CalciteTests.DRUID_SCHEMA_NAME,
+ new CalciteRulesManager(ImmutableSet.of()),
+ CalciteTests.createJoinableFactoryWrapper(),
+ CatalogResolver.NULL_RESOLVER,
+ new AuthConfig()
+ );
+
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(
+ engine,
+ sql,
+ ImmutableMap.of(
+ "inSubQueryThreshold",
+ inSubQueryThreshold,
+ "useCache",
+ false
+ )
+ )) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] planResult = resultSequence.toList().get(0);
+ log.info("Query plan:\n" +
+ jsonMapper.writerWithDefaultPrettyPrinter()
+ .writeValueAsString(jsonMapper.readValue((String) planResult[0], List.class))
+ );
+ }
+ catch (JsonMappingException e) {
+ throw new RuntimeException(e);
+ }
+ catch (JsonProcessingException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @TearDown(Level.Trial)
+ public void tearDown() throws Exception
+ {
+ closer.close();
+ }
+
+ @BenchmarkMode(Mode.AverageTime)
+ @OutputTimeUnit(TimeUnit.MILLISECONDS)
+ public void queryInSql(Blackhole blackhole)
+ {
+ final Map<String, Object> context = ImmutableMap.of(
+ "inSubQueryThreshold", inSubQueryThreshold, "useCache", false
+ );
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(engine, sql, context)) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in);
+ blackhole.consume(lastRow);
+ }
+ }
+
+ @Benchmark
+ @BenchmarkMode(Mode.AverageTime)
+ @OutputTimeUnit(TimeUnit.MILLISECONDS)
+ public void queryEqualOrInSql(Blackhole blackhole)
+ {
+ final Map<String, Object> context = ImmutableMap.of(
+ "inSubQueryThreshold", inSubQueryThreshold, "useCache", false
+ );
+ StringBuilder sqlBuilder = new StringBuilder().append(
+ "explain plan for select long1 from foo where string1 = '7' or long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(engine, sql, context)) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in);
+ blackhole.consume(lastRow);
+ }
+ }
+
+
+ @Benchmark
+ @BenchmarkMode(Mode.AverageTime)
+ @OutputTimeUnit(TimeUnit.MILLISECONDS)
+ public void queryMultiEqualOrInSql(Blackhole blackhole)
+ {
+ final Map<String, Object> context = ImmutableMap.of(
+ "inSubQueryThreshold", inSubQueryThreshold, "useCache", false
+ );
+ StringBuilder sqlBuilder = new StringBuilder().append(
+ "explain plan for select long1 from foo where string1 = '7' or string1 = '8' or long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
+
+ try (final DruidPlanner planner = plannerFactory.createPlannerForTesting(engine, sql, context)) {
+ final PlannerResult plannerResult = planner.plan();
+ final Sequence<Object[]> resultSequence = plannerResult.run().getResults();
+ final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in);
+ blackhole.consume(lastRow);
+ }
Review Comment:
nit: this seems like a common pattern in all new tests - it could be extracted into a method
##########
benchmarks/src/test/java/org/apache/druid/benchmark/query/InPlanningBenchmark.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark.query;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.impl.DimensionSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.math.expr.ExpressionProcessing;
+import org.apache.druid.query.DruidProcessingConfig;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.segment.AutoTypeColumnSchema;
+import org.apache.druid.segment.IndexSpec;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.generator.GeneratorBasicSchemas;
+import org.apache.druid.segment.generator.GeneratorSchemaInfo;
+import org.apache.druid.segment.generator.SegmentGenerator;
+import org.apache.druid.segment.transform.TransformSpec;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.sql.calcite.planner.CalciteRulesManager;
+import org.apache.druid.sql.calcite.planner.CatalogResolver;
+import org.apache.druid.sql.calcite.planner.DruidPlanner;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.planner.PlannerResult;
+import org.apache.druid.sql.calcite.run.SqlEngine;
+import org.apache.druid.sql.calcite.schema.DruidSchemaCatalog;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import javax.annotation.Nullable;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Benchmark that tests various SQL queries.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+public class InPlanningBenchmark
+{
+ private static final Logger log = new Logger(SqlExpressionBenchmark.class);
+
+ static {
+ NullHandling.initializeForTests();
+ ExpressionProcessing.initializeForTests();
+ }
+
+ private static final DruidProcessingConfig PROCESSING_CONFIG = new DruidProcessingConfig()
+ {
+ @Override
+ public int intermediateComputeSizeBytes()
+ {
+ return 512 * 1024 * 1024;
+ }
+
+ @Override
+ public int getNumMergeBuffers()
+ {
+ return 3;
+ }
+
+ @Override
+ public int getNumThreads()
+ {
+ return 1;
+ }
+
+ @Override
+ public String getFormatString()
+ {
+ return "benchmarks-processing-%s";
+ }
+ };
+
+ @Param({"500000"})
+ private int rowsPerSegment;
+
+ @Param({"0", "2147483647"})
+ private int inSubQueryThreshold;
+
+ @Param({
+ "auto"
+ })
+ private String schema;
+
+ @Param({
+ "1", "10", "100", "1000", "10000", "100000", "1000000"
+ })
+ private Integer inClauseExprCount;
+ private SqlEngine engine;
+ @Nullable
+ private PlannerFactory plannerFactory;
+ private Closer closer = Closer.create();
+
+ @Setup(Level.Trial)
+ public void setup()
+ {
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("in-testbench");
+
+ final DataSegment dataSegment = DataSegment.builder()
+ .dataSource("foo")
+ .interval(schemaInfo.getDataInterval())
+ .version("1")
+ .shardSpec(new LinearShardSpec(0))
+ .size(0)
+ .build();
+
+ final PlannerConfig plannerConfig = new PlannerConfig();
+
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+ log.info(
+ "Starting benchmark setup using cacheDir[%s], rows[%,d], schema[%s].",
+ segmentGenerator.getCacheDir(),
+ rowsPerSegment,
+ schema
+ );
+ final QueryableIndex index;
+ if ("auto".equals(schema)) {
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ index = segmentGenerator.generate(
+ dataSegment,
+ schemaInfo,
+ DimensionsSpec.builder().setDimensions(columnSchemas).build(),
+ TransformSpec.NONE,
+ IndexSpec.DEFAULT,
+ Granularities.NONE,
+ rowsPerSegment
+ );
+ } else {
+ index = segmentGenerator.generate(dataSegment, schemaInfo, Granularities.NONE, rowsPerSegment);
+ }
+
+ final QueryRunnerFactoryConglomerate conglomerate = QueryStackTests.createQueryRunnerFactoryConglomerate(
+ closer,
+ PROCESSING_CONFIG
+ );
+
+ final SpecificSegmentsQuerySegmentWalker walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(
+ dataSegment,
+ index
+ );
+ closer.register(walker);
+ final ObjectMapper jsonMapper = CalciteTests.getJsonMapper();
+ final DruidSchemaCatalog rootSchema =
+ CalciteTests.createMockRootSchema(conglomerate, walker, plannerConfig, AuthTestUtils.TEST_AUTHORIZER_MAPPER);
+ engine = CalciteTests.createMockSqlEngine(walker, conglomerate);
+ plannerFactory = new PlannerFactory(
+ rootSchema,
+ CalciteTests.createOperatorTable(),
+ CalciteTests.createExprMacroTable(),
+ plannerConfig,
+ AuthTestUtils.TEST_AUTHORIZER_MAPPER,
+ jsonMapper,
+ CalciteTests.DRUID_SCHEMA_NAME,
+ new CalciteRulesManager(ImmutableSet.of()),
+ CalciteTests.createJoinableFactoryWrapper(),
+ CatalogResolver.NULL_RESOLVER,
+ new AuthConfig()
+ );
+
+ StringBuilder sqlBuilder = new StringBuilder().append("explain plan for select long1 from foo where long1 in (");
+ IntStream.range(1, inClauseExprCount - 1).forEach(i -> sqlBuilder.append(i).append(","));
+ sqlBuilder.append(inClauseExprCount).append(")");
+ final String sql = sqlBuilder.toString();
Review Comment:
note: this seems like the same as one of the tests; couldn't you reuse the test's code to do this?
note: I also wonder if this is necessary - as benchmark framework also runs warmups .. or the sole reason to have this is to show the query plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org