You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "nozjkoitop (via GitHub)" <gi...@apache.org> on 2026/02/10 14:40:07 UTC
[PR] introduced configurable index disabling for virtual columns (druid)
nozjkoitop opened a new pull request, #19004:
URL: https://github.com/apache/druid/pull/19004
### Description
In some of our environments we observed a significant query performance regression after upgrading to Druid 29.0. This regression appears to be related to the virtual column bitmap indexing optimization introduced in https://github.com/apache/druid/pull/15585 and https://github.com/apache/druid/pull/15633.
While the change improves performance for many cases by enabling bitmap index creation for expression-based virtual columns, we found that in certain workloads the index computation becomes unexpectedly expensive. In our deployments, building these indices accounted for more than 90% of the CPU usage on Historical nodes during query execution, leading to severe degradation in overall query latency.
For one representative workload, the average query runtime increased from 10.01 seconds to 150.18 seconds compared to earlier versions. Once expression-based virtual column bitmap index creation was disabled, the regression disappeared and query performance returned to expected levels.
This PR addresses this issue by preventing the optimization from triggering in scenarios where the cost of index computation outweighs its benefit, avoiding major regressions in affected environments.
Added a new query context parameter, `maxVirtualColumnsForBitmapIndexing` (default: `Integer.MAX_VALUE`), which sets the virtual-column count threshold beyond which Druid stops using bitmap indexes for filters on virtual columns.
#### Release note
Adds a safeguard to skip virtual-column bitmap indexing when it is likely to be counterproductive, falling back to non-indexed filtering to preserve expected performance.
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3898712907
added a case with broad, expression-heavy virtual-column filters
there were likely other improvements in later versions, but I still saw ~12% better performance with indexing disabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899115589
Interesting, although I was using lz4 compression + FRONT_CODED_16_V1 encoding, ur results surprised me :) lemme give it another try
Could u share the JMH flags used please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899033603
hmm, I still show the added benchmark query as faster with indexes, especially for vectorized processing. I didn't pull your branch, just used apache master with the query added and to test no indexes modified [this line ExpressionVirtualColumn](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/virtual/ExpressionVirtualColumn.java#L274) to return `NoIndexesColumnIndexSupplier.getInstance()`
current master:
```
Benchmark (complexCompression) (deferExpressionDimensions) (jsonObjectStorageEncoding) (query) (rowsPerSegment) (schemaType) (storageType) (stringEncoding) (vectorize) Mode Cnt Score Error Units
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 437.811 ± 57.658 ms/op
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 371.432 ± 2.740 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 417.081 ± 10.159 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 368.717 ± 8.266 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 417.167 ± 4.085 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 369.220 ± 10.618 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 417.864 ± 13.421 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 344.602 ± 9.518 ms/op
```
modified so that no indexes are used:
```
Benchmark (complexCompression) (deferExpressionDimensions) (jsonObjectStorageEncoding) (query) (rowsPerSegment) (schemaType) (storageType) (stringEncoding) (vectorize) Mode Cnt Score Error Units
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 505.174 ± 7.250 ms/op
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 1165.574 ± 37.120 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 501.502 ± 33.384 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 1160.577 ± 43.142 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 501.388 ± 24.289 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 1198.882 ± 57.135 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP UTF8 false avgt 5 508.851 ± 4.995 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP UTF8 force avgt 5 1206.142 ± 37.526 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899870911
re front-coding, i think i see what is going on, [the `ExpressionPredicateIndexSupplier` that is computing the indexes](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/math/expr/ExpressionPredicateIndexSupplier.java#L262) is scanning the dictionary in order to find all of the values that match, but is using random access to get the values which for front coding means seeking in the bucket to the position we actually need which is basically where all of the cost of using it is (in exchange for the smaller sizes). I suspect when it is not using the indexes that perhaps the matcher must be able to rule out a match earlier and so fewer overall calls to [`FrontCodedIndexed.get`](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/data/FrontCodedIndexed.java#L208) are happening.
I think since we are checking every dictionary value, it would be a lot more chill for front-coding if use used the dictionary iterator instead of calling get, it needs to be exposed on `DictionaryEncodedValueIndex` so that the expression predicate indexes could use it. Will look into this :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop closed pull request #19004: Configurable index disabling for virtual columns
URL: https://github.com/apache/druid/pull/19004
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899645841
a neat, I do show front-coded as a bit slower for non-vectorized, which is curious, will look into that a bit more
with index:
```
Benchmark (complexCompression) (deferExpressionDimensions) (jsonObjectStorageEncoding) (query) (rowsPerSegment) (schemaType) (storageType) (stringEncoding) (vectorize) Mode Cnt Score Error Units
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 522.387 ± 22.942 ms/op
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 501.122 ± 17.559 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 547.506 ± 15.055 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 446.650 ± 5.308 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 572.099 ± 67.823 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 499.534 ± 19.926 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 549.607 ± 25.846 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 496.660 ± 16.439 ms/op
```
without:
```
Benchmark (complexCompression) (deferExpressionDimensions) (jsonObjectStorageEncoding) (query) (rowsPerSegment) (schemaType) (storageType) (stringEncoding) (vectorize) Mode Cnt Score Error Units
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 516.153 ± 18.747 ms/op
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 1154.914 ± 34.838 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 438.316 ± 24.513 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 1204.622 ± 57.132 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 431.339 ± 23.974 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 1199.698 ± 34.916 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 518.188 ± 19.511 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 1235.943 ± 38.730 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3886133551
> Thanks for the references, unfortunately this still reproduces for us on 31.
To put numbers behind this, i can try adding a benchmark that shows the impact
that would be great, there is [SqlExpressionBenchmark](https://github.com/apache/druid/blob/master/benchmarks/src/test/java/org/apache/druid/benchmark/query/SqlExpressionBenchmark.java) which uses a [data generator to have a collection of columns with various types and data distributions](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/generator/GeneratorBasicSchemas.java#L331) maybe you can add something that looks approximately like the problem you're running into.
I would like to try to see if this is something we can improve without adding a new manual parameter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899933482
ah yea, it is totally that, using the iterator improves the measurement on using the indexes quite a lot
with indexes using iterator:
```
Benchmark (complexCompression) (deferExpressionDimensions) (jsonObjectStorageEncoding) (query) (rowsPerSegment) (schemaType) (storageType) (stringEncoding) (vectorize) Mode Cnt Score Error Units
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 428.333 ± 14.320 ms/op
SqlExpressionBenchmark.querySql NONE singleString SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 364.073 ± 5.671 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 423.951 ± 12.710 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidth SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 371.926 ± 5.133 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 424.357 ± 10.445 ms/op
SqlExpressionBenchmark.querySql NONE fixedWidthNonNumeric SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 419.708 ± 71.678 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 false avgt 5 444.724 ± 112.962 ms/op
SqlExpressionBenchmark.querySql NONE always SMILE 61 1500000 explicit MMAP FRONT_CODED_16_V1 force avgt 5 373.843 ± 8.409 ms/op
```
thanks for bringing this to attention :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899497679
thanks, i'll return it with matching params and let u know about results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3907191334
appreciate the input, that's a clean fix, nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3900077888
opened #19023 to help perf for front-coding + expression indexes (and maybe a few other things)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19004:
URL: https://github.com/apache/druid/pull/19004#discussion_r3141623611
##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorHolder.java:
##########
@@ -672,14 +676,19 @@ private CursorResources(
Interval interval,
@Nullable Filter filter,
boolean cursorAutoArrangeFilters,
- @Nullable QueryMetrics<? extends Query<?>> metrics
+ @Nullable QueryMetrics<? extends Query<?>> metrics,
+ boolean disableVirtualColumnBitmapIndexes
)
{
this.closer = Closer.create();
this.columnCache = new ColumnCache(index, virtualColumns, closer);
this.timeBoundaryInspector = timeBoundaryInspector;
try {
this.numRows = index.getNumRows();
+ final ColumnIndexSelector indexSelectorForFilters =
Review Comment:
[P2] Search queries still bypass this virtual-column index guard
This only wraps the selector used by cursor filter bundles, but `SearchQuery` has a separate index-only path in `UseIndexesStrategy` that still constructs plain `ColumnCache` selectors for `partitionDimensionList`, `makeTimeFilteredBitmap`, and `IndexOnlyExecutor`. A search query with `maxVirtualColumnsForBitmapIndexing` set low and a virtual search dimension or virtual-column filter can still choose and execute the bitmap-index plan, so the new query context is not honored for one of the query types that accepts virtual columns. Please apply the same disabling selector in the search index strategy, or document/limit the parameter to cursor-backed query paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-4272171182
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-4321466881
yeah, there is no need in this one anymore, closing it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3899366423
```
DRUID_BENCHMARK_CACHE_DIR=./tmp java --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -server -jar benchmarks/target/benchmarks.jar -p query=61 -p stringEncoding=UTF8 -p complexCompression=NONE -p schemaType=explicit -p storageType=MMAP org.apache.druid.benchmark.query.SqlExpressionBenchmark
```
I'm using java 21 on a m1 mac for additional context.
>Interesting, although I was using lz4 compression + FRONT_CODED_16_V1 encoding
i think by default everything here would be using lz4 since that parameter is only for complex columns to measure #16863 and has no impact if not using complex columns, but I will try with front coding though, since significantly slower perf there would also be quite interesting to look into
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3881665177
>In some of our environments we observed a significant query performance regression after upgrading to Druid 29.0. This regression appears to be related to the virtual column bitmap indexing optimization introduced in https://github.com/apache/druid/pull/15585 and https://github.com/apache/druid/pull/15633.
have you tried in versions newer than 29? #15838 was in 30, and #17055/#17125 in 31, which should make it so that expression indexes are only used in cases where there would be less work to do than a full scan (which has to compute the expression value and perform the match for every row).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-3885035022
Thanks for the references, unfortunately this still reproduces for us on 31.
To put numbers behind this, i can try adding a benchmark that shows the impact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Configurable index disabling for virtual columns (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on PR #19004:
URL: https://github.com/apache/druid/pull/19004#issuecomment-4321381651
Is this PR still being pursued now that https://github.com/apache/druid/pull/19023 is in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org