You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Akshat-Jain (via GitHub)" <gi...@apache.org> on 2024/10/18 11:29:41 UTC
[PR] Add benchmark suite for MSQ window functions (druid)
Akshat-Jain opened a new pull request, #17377:
URL: https://github.com/apache/druid/pull/17377
### Description
This PR adds a benchmark suite for MSQ window function queries.
A sample run on my local gave the following results:
```
Benchmark (maxNumTasks) (rowsPerSegment) Mode Cnt Score Error Units
MSQWindowFunctionsBenchmark.windowWithoutGroupBy 2 20000000 avgt 5 96681.604 ± 2425.579 ms/op
MSQWindowFunctionsBenchmark.windowWithoutGroupBy 5 20000000 avgt 5 94676.305 ± 4012.108 ms/op
MSQWindowFunctionsBenchmark.windowWithoutSorting 2 20000000 avgt 5 17515.970 ± 498.066 ms/op
MSQWindowFunctionsBenchmark.windowWithoutSorting 5 20000000 avgt 5 15996.262 ± 1552.218 ms/op
MSQWindowFunctionsBenchmark.multipleWindows 2 20000000 avgt 5 63215.499 ± 4722.604 ms/op
MSQWindowFunctionsBenchmark.multipleWindows 5 20000000 avgt 5 69287.847 ± 4985.326 ms/op
MSQWindowFunctionsBenchmark.windowWithHighCardinalityPartitionBy 2 20000000 avgt 5 69469.122 ± 3016.019 ms/op
MSQWindowFunctionsBenchmark.windowWithHighCardinalityPartitionBy 5 20000000 avgt 5 70951.896 ± 4343.354 ms/op
MSQWindowFunctionsBenchmark.windowWithLowCardinalityPartitionBy 2 20000000 avgt 5 507.584 ± 600.999 ms/op
MSQWindowFunctionsBenchmark.windowWithLowCardinalityPartitionBy 5 20000000 avgt 5 413.795 ± 38.195 ms/op
MSQWindowFunctionsBenchmark.windowWithSorting 2 20000000 avgt 5 16682.792 ± 239.561 ms/op
MSQWindowFunctionsBenchmark.windowWithSorting 5 20000000 avgt 5 16422.890 ± 225.643 ms/op
```
<hr>
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1820079453
##########
sql/src/test/java/org/apache/druid/sql/calcite/QueryTestRunner.java:
##########
@@ -772,8 +772,11 @@ public void run()
public QueryResults resultsOnly()
{
- ExecuteQuery execStep = (ExecuteQuery) runSteps.get(0);
- execStep.run();
+ for (QueryRunStep runStep : runSteps) {
+ runStep.run();
+ }
+
+ BaseExecuteQuery execStep = (BaseExecuteQuery) runSteps.get(runSteps.size() - 1);
Review Comment:
`queryTestBuilder.results()` didn't support MSQ apparently, since it only had the execute step, and not the ExtractResultsFactory step.
So, without this change, instead of getting the query results, we were getting the query ID. The ExtractResultsFactory step (added as a custom runner when declaring QueryTestBuilder) takes the query ID, and contacts the overlord client, and fetches the actual results.
> (and not others)?
In the regular MSQ tests, we do `testBuilder().run()` which handled both steps already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818369522
##########
benchmarks/pom.xml:
##########
@@ -217,6 +227,13 @@
<version>${project.parent.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.druid.extensions</groupId>
+ <artifactId>druid-multi-stage-query</artifactId>
Review Comment:
The existing dependency pulls in the main classes, whereas this one pulls in the test classes (the difference is in the `<classifier>tests</classifier>` part). Hence both are needed.
There doesn't seem to be a way to "combine" the two dependencies into one, hence had to add the test dependency separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818421860
##########
sql/src/test/java/org/apache/druid/sql/calcite/util/TestDataBuilder.java:
##########
@@ -1014,6 +1039,40 @@ public static QueryableIndex getQueryableIndexForDrillDatasource(String datasour
.buildMMappedIndex();
}
+ public static QueryableIndex getQueryableIndexForBenchmarkDatasource()
+ {
+ if (QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE == null) {
+ throw new RuntimeException("Queryable index was not populated for benchmark datasource.");
+ }
+ return QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE;
+ }
+
+ public static void makeQueryableIndexForBenchmarkDatasource(Closer closer, int rowsPerSegment)
+ {
+ if (closer == null) {
+ throw new RuntimeException("Closer not supplied for generating segments, exiting.");
+ }
+
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("basic");
+ final DataSegment dataSegment = schemaInfo.makeSegmentDescriptor(CalciteTests.BENCHMARK_DATASOURCE);
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE = segmentGenerator.generate(
Review Comment:
This segment generation logic already exists. For example, it's used in `SqlWindowFunctionsBenchmark` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "adarshsanjeev (via GitHub)" <gi...@apache.org>.
adarshsanjeev commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1820065415
##########
sql/src/test/java/org/apache/druid/sql/calcite/QueryTestRunner.java:
##########
@@ -772,8 +772,11 @@ public void run()
public QueryResults resultsOnly()
{
- ExecuteQuery execStep = (ExecuteQuery) runSteps.get(0);
- execStep.run();
+ for (QueryRunStep runStep : runSteps) {
+ runStep.run();
+ }
+
+ BaseExecuteQuery execStep = (BaseExecuteQuery) runSteps.get(runSteps.size() - 1);
Review Comment:
I'm not very familiar with this, why is the change required to run all runsteps for this benchmark (and not others)?
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/CalciteMSQTestsHelper.java:
##########
@@ -416,6 +416,9 @@ protected static Supplier<ResourceHolder<CompleteSegment>> getSupplierForSegment
case CalciteTests.T_ALL_TYPE_PARQUET:
index = TestDataBuilder.getQueryableIndexForDrillDatasource(segmentId.getDataSource(), tempFolderProducer.apply("tmpDir"));
break;
+ case CalciteTests.BENCHMARK_DATASOURCE:
Review Comment:
It does look a bit strange to have details of a datasource used by a single test here. This class would probably explode in size if this was the case for more tests. We can refactor it in a followup though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818370716
##########
extensions-core/multi-stage-query/pom.xml:
##########
@@ -352,6 +352,17 @@
</excludes>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
Review Comment:
Yeah, the MSQ query benchmark implementation is relying on the MSQ module's test classes. This change is needed so that the test JAR is generated, for use by the benchmark module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818370716
##########
extensions-core/multi-stage-query/pom.xml:
##########
@@ -352,6 +352,17 @@
</excludes>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
Review Comment:
Yeah, the MSQ query benchmark implementation is relying on the MSQ module's test classes. This change is needed so that the test JAR is generated, for use of the benchmark module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818373338
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/CalciteMSQTestsHelper.java:
##########
@@ -416,6 +416,9 @@ protected static Supplier<ResourceHolder<CompleteSegment>> getSupplierForSegment
case CalciteTests.T_ALL_TYPE_PARQUET:
index = TestDataBuilder.getQueryableIndexForDrillDatasource(segmentId.getDataSource(), tempFolderProducer.apply("tmpDir"));
break;
+ case CalciteTests.BENCHMARK_DATASOURCE:
Review Comment:
> Is there scope in reusing this datasource for anything else? If so, it could be renamed to a more general name.
No, there isn't any scope in re-using this for anything else.
> If not, I'm wondering if there is a better way to override this just for the one test.
I'm not sure if that's doable, it's not trivial to achieve atleast.
With that said, we did it in a similar way when we added datasources for drill test suite (the `case` statements just above the added line of code) - so if we really want to, I feel it could be a follow-up patch for both drill and benchmark datasources. But I also think it doesn't add any value to over-engineer this, since this change doesn't affect any other test. Hope that makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "adarshsanjeev (via GitHub)" <gi...@apache.org>.
adarshsanjeev commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818342836
##########
extensions-core/multi-stage-query/pom.xml:
##########
@@ -352,6 +352,17 @@
</excludes>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
Review Comment:
Is this change needed for running benchmarks?
##########
sql/src/test/java/org/apache/druid/sql/calcite/util/TestDataBuilder.java:
##########
@@ -1014,6 +1039,40 @@ public static QueryableIndex getQueryableIndexForDrillDatasource(String datasour
.buildMMappedIndex();
}
+ public static QueryableIndex getQueryableIndexForBenchmarkDatasource()
+ {
+ if (QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE == null) {
+ throw new RuntimeException("Queryable index was not populated for benchmark datasource.");
+ }
+ return QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE;
+ }
+
+ public static void makeQueryableIndexForBenchmarkDatasource(Closer closer, int rowsPerSegment)
+ {
+ if (closer == null) {
+ throw new RuntimeException("Closer not supplied for generating segments, exiting.");
+ }
+
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("basic");
+ final DataSegment dataSegment = schemaInfo.makeSegmentDescriptor(CalciteTests.BENCHMARK_DATASOURCE);
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE = segmentGenerator.generate(
Review Comment:
What are the actual rows that are added to this datasource?
##########
sql/src/test/java/org/apache/druid/sql/calcite/SqlTestFrameworkConfig.java:
##########
@@ -301,7 +301,7 @@ public void beforeEach(ExtensionContext context)
setConfig(context);
}
- private void setConfig(ExtensionContext context)
+ public void setConfig(ExtensionContext context)
Review Comment:
nit: this could be renamed as it does not look like a setter, now that you have added an actual setter below.
##########
benchmarks/pom.xml:
##########
@@ -217,6 +227,13 @@
<version>${project.parent.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.druid.extensions</groupId>
+ <artifactId>druid-multi-stage-query</artifactId>
Review Comment:
This dependency is already present in this pom, could is this needed?
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/CalciteMSQTestsHelper.java:
##########
@@ -416,6 +416,9 @@ protected static Supplier<ResourceHolder<CompleteSegment>> getSupplierForSegment
case CalciteTests.T_ALL_TYPE_PARQUET:
index = TestDataBuilder.getQueryableIndexForDrillDatasource(segmentId.getDataSource(), tempFolderProducer.apply("tmpDir"));
break;
+ case CalciteTests.BENCHMARK_DATASOURCE:
Review Comment:
Is there scope in reusing this datasource for anything else? If so, it could be renamed to a more general name. If not, I'm wondering if there is a better way to override this just for the one test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818424902
##########
sql/src/test/java/org/apache/druid/sql/calcite/util/TestDataBuilder.java:
##########
@@ -1014,6 +1039,40 @@ public static QueryableIndex getQueryableIndexForDrillDatasource(String datasour
.buildMMappedIndex();
}
+ public static QueryableIndex getQueryableIndexForBenchmarkDatasource()
+ {
+ if (QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE == null) {
+ throw new RuntimeException("Queryable index was not populated for benchmark datasource.");
+ }
+ return QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE;
+ }
+
+ public static void makeQueryableIndexForBenchmarkDatasource(Closer closer, int rowsPerSegment)
+ {
+ if (closer == null) {
+ throw new RuntimeException("Closer not supplied for generating segments, exiting.");
+ }
+
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("basic");
+ final DataSegment dataSegment = schemaInfo.makeSegmentDescriptor(CalciteTests.BENCHMARK_DATASOURCE);
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE = segmentGenerator.generate(
Review Comment:
Oh, I guess you meant where `QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE` is initialised.
`TestDataBuilder#makeQueryableIndexForBenchmarkDatasource` gets called from `MSQWindowFunctionsBenchmark#setup`.
Hope this clarifies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "adarshsanjeev (via GitHub)" <gi...@apache.org>.
adarshsanjeev commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818418699
##########
sql/src/test/java/org/apache/druid/sql/calcite/util/TestDataBuilder.java:
##########
@@ -1014,6 +1039,40 @@ public static QueryableIndex getQueryableIndexForDrillDatasource(String datasour
.buildMMappedIndex();
}
+ public static QueryableIndex getQueryableIndexForBenchmarkDatasource()
+ {
+ if (QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE == null) {
+ throw new RuntimeException("Queryable index was not populated for benchmark datasource.");
+ }
+ return QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE;
+ }
+
+ public static void makeQueryableIndexForBenchmarkDatasource(Closer closer, int rowsPerSegment)
+ {
+ if (closer == null) {
+ throw new RuntimeException("Closer not supplied for generating segments, exiting.");
+ }
+
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("basic");
+ final DataSegment dataSegment = schemaInfo.makeSegmentDescriptor(CalciteTests.BENCHMARK_DATASOURCE);
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE = segmentGenerator.generate(
Review Comment:
Where is this initialized? I don't see it in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe merged PR #17377:
URL: https://github.com/apache/druid/pull/17377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818376807
##########
sql/src/test/java/org/apache/druid/sql/calcite/util/TestDataBuilder.java:
##########
@@ -1014,6 +1039,40 @@ public static QueryableIndex getQueryableIndexForDrillDatasource(String datasour
.buildMMappedIndex();
}
+ public static QueryableIndex getQueryableIndexForBenchmarkDatasource()
+ {
+ if (QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE == null) {
+ throw new RuntimeException("Queryable index was not populated for benchmark datasource.");
+ }
+ return QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE;
+ }
+
+ public static void makeQueryableIndexForBenchmarkDatasource(Closer closer, int rowsPerSegment)
+ {
+ if (closer == null) {
+ throw new RuntimeException("Closer not supplied for generating segments, exiting.");
+ }
+
+ final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("basic");
+ final DataSegment dataSegment = schemaInfo.makeSegmentDescriptor(CalciteTests.BENCHMARK_DATASOURCE);
+ final SegmentGenerator segmentGenerator = closer.register(new SegmentGenerator());
+
+ List<DimensionSchema> columnSchemas = schemaInfo.getDimensionsSpec()
+ .getDimensions()
+ .stream()
+ .map(x -> new AutoTypeColumnSchema(x.getName(), null))
+ .collect(Collectors.toList());
+ QUERYABLE_INDEX_FOR_BENCHMARK_DATASOURCE = segmentGenerator.generate(
Review Comment:
Pasting the row signature, and 5 sample rows:
```
signature = {__time:LONG, dimSequential:STRING, dimZipf:STRING, dimUniform:STRING, dimSequentialHalfNull:STRING, dimMultivalEnumerated:ARRAY<STRING>, dimMultivalEnumerated2:ARRAY<STRING>, dimMultivalSequentialWithNulls:ARRAY<STRING>, dimHyperUnique:STRING, dimNull:STRING, hyper:COMPLEX<hyperUnique>, maxLongUniform:LONG, minFloatZipf:DOUBLE, rows:LONG, sumFloatNormal:DOUBLE, sumLongSequential:LONG}
new Object[]{946689120000L, "0", "49", "85847", null, "[\"Baz\",\"Baz\",\"World\",\"Baz\"]", "[null,null,\"Orange\"]", null, "0", null, "\"AQAAAQAAAAKkgA==\"", 429L, 347.0D, 1L, 5000.0D, 0L},
new Object[]{946693440000L, "1", "49", "85832", null, "[\"World\",\"Baz\",\"World\",\"Baz\"]", "[null,\"Orange\",null]", null, "1", null, "\"AQAAAQAAAAFREA==\"", 429L, 346.0D, 1L, 5000.0D, 1L},
new Object[]{946697760000L, "2", "3", "21503", "0", "[\"Baz\",\"Foo\",\"Hello\",\"Bar\"]", "[\"Orange\",null,null]", "[\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\"]", "2", null, "\"AQAAAQAAAAOzAg==\"", 107L, 3.0D, 1L, 5000.0D, 2L},
new Object[]{946702080000L, "3", "91", "98132", "1", "[\"Bar\",\"Foo\",\"Hello\",\"Baz\"]", "[\"Xylophone\",\"Apple\",\"Corundum\"]", "[\"9\",\"10\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\"]", "3", null, "\"AQAAAQAAAAGdQA==\"", 490L, 869.0D, 1L, 5000.0D, 3L},
new Object[]{946706400000L, "4", "7", "44371", "2", "[\"Bar\",\"World\",\"Baz\",\"Baz\"]", "[\"Corundum\",\"Xylophone\",\"Apple\"]", "[\"7\",\"8\",\"9\",\"10\",\"1\",\"2\",\"3\",\"4\"]", "4", null, "\"AQAAAQAAAALwEA==\"", 221L, 16.0D, 1L, 5000.0D, 4L}
```
Does this help? Let me know if there's any specific aspect that you wanted to know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Add benchmark suite for MSQ window functions (druid)
Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #17377:
URL: https://github.com/apache/druid/pull/17377#discussion_r1818379733
##########
sql/src/test/java/org/apache/druid/sql/calcite/SqlTestFrameworkConfig.java:
##########
@@ -301,7 +301,7 @@ public void beforeEach(ExtensionContext context)
setConfig(context);
}
- private void setConfig(ExtensionContext context)
+ public void setConfig(ExtensionContext context)
Review Comment:
Fair point. Have renamed it to `makeConfigFromContext`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org