You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/08/02 04:34:02 UTC
[GitHub] [druid] clintropolis opened a new pull request, #12846: specialized FixedIndexed implementations for java value types
clintropolis opened a new pull request, #12846:
URL: https://github.com/apache/druid/pull/12846
### Description
Adds specialized implementations for java long, double, and int value type implementations of `FixedIndexed`, which is used by the nested data columns added in #12753.
While not entirely attributable to this PR (the range filtering tests owe that to ), repeating the benchmarks done in show
improvement:
```
SELECT SUM(long1) FROM foo
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 0 5000000 false avgt 5 36.711 ± 0.917 ms/op
SqlNestedDataBenchmark.querySql 0 5000000 force avgt 5 15.587 ± 0.276 ms/op
SqlNestedDataBenchmark.querySql 1 5000000 false avgt 5 39.224 ± 0.870 ms/op
SqlNestedDataBenchmark.querySql 1 5000000 force avgt 5 15.877 ± 0.440 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 0 5000000 false avgt 5 37.672 ± 1.136 ms/op
SqlNestedDataBenchmark.querySql 0 5000000 force avgt 5 15.687 ± 0.497 ms/op
SqlNestedDataBenchmark.querySql 1 5000000 false avgt 5 39.437 ± 0.690 ms/op
SqlNestedDataBenchmark.querySql 1 5000000 force avgt 5 15.978 ± 0.587 ms/op
SELECT SUM(long1), SUM(long2) FROM foo
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT)) FROM foo
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 2 5000000 false avgt 5 63.805 ± 1.036 ms/op
SqlNestedDataBenchmark.querySql 2 5000000 force avgt 5 30.381 ± 1.201 ms/op
SqlNestedDataBenchmark.querySql 3 5000000 false avgt 5 66.660 ± 0.806 ms/op
SqlNestedDataBenchmark.querySql 3 5000000 force avgt 5 30.341 ± 1.124 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 2 5000000 false avgt 5 63.825 ± 1.228 ms/op
SqlNestedDataBenchmark.querySql 2 5000000 force avgt 5 30.955 ± 0.769 ms/op
SqlNestedDataBenchmark.querySql 3 5000000 false avgt 5 67.277 ± 0.928 ms/op
SqlNestedDataBenchmark.querySql 3 5000000 force avgt 5 30.860 ± 1.023 ms/op
SELECT SUM(long1), SUM(long2), SUM(double3) FROM foo
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 4 5000000 false avgt 5 78.570 ± 1.657 ms/op
SqlNestedDataBenchmark.querySql 4 5000000 force avgt 5 37.777 ± 1.295 ms/op
SqlNestedDataBenchmark.querySql 5 5000000 false avgt 5 82.672 ± 1.010 ms/op
SqlNestedDataBenchmark.querySql 5 5000000 force avgt 5 37.887 ± 0.802 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 4 5000000 false avgt 5 80.173 ± 1.342 ms/op
SqlNestedDataBenchmark.querySql 4 5000000 force avgt 5 38.272 ± 0.589 ms/op
SqlNestedDataBenchmark.querySql 5 5000000 false avgt 5 84.370 ± 1.275 ms/op
SqlNestedDataBenchmark.querySql 5 5000000 force avgt 5 38.541 ± 1.137 ms/op
SELECT string1, SUM(long1) FROM foo GROUP BY 1 ORDER BY 2,
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo GROUP BY 1 ORDER BY 2,
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 6 5000000 false avgt 5 269.560 ± 1.454 ms/op
SqlNestedDataBenchmark.querySql 6 5000000 force avgt 5 157.090 ± 4.058 ms/op
SqlNestedDataBenchmark.querySql 7 5000000 false avgt 5 373.162 ± 2.871 ms/op
SqlNestedDataBenchmark.querySql 7 5000000 force avgt 5 195.213 ± 1.993 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 6 5000000 false avgt 5 234.328 ± 5.505 ms/op
SqlNestedDataBenchmark.querySql 6 5000000 force avgt 5 155.711 ± 5.286 ms/op
SqlNestedDataBenchmark.querySql 7 5000000 false avgt 5 383.741 ± 4.796 ms/op
SqlNestedDataBenchmark.querySql 7 5000000 force avgt 5 195.051 ± 6.225 ms/op
SELECT string1, SUM(long1), SUM(double3) FROM foo GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo GROUP BY 1 ORDER BY 2
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 8 5000000 false avgt 5 251.743 ± 6.438 ms/op
SqlNestedDataBenchmark.querySql 8 5000000 force avgt 5 172.322 ± 14.814 ms/op
SqlNestedDataBenchmark.querySql 9 5000000 false avgt 5 417.454 ± 21.276 ms/op
SqlNestedDataBenchmark.querySql 9 5000000 force avgt 5 215.228 ± 9.304 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 8 5000000 false avgt 5 302.745 ± 6.738 ms/op
SqlNestedDataBenchmark.querySql 8 5000000 force avgt 5 168.410 ± 1.750 ms/op
SqlNestedDataBenchmark.querySql 9 5000000 false avgt 5 459.633 ± 5.099 ms/op
SqlNestedDataBenchmark.querySql 9 5000000 force avgt 5 208.978 ± 1.130 ms/op
SELECT SUM(long1) FROM foo WHERE string1 = '10000' OR string1 = '1000'
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.string1') = '10000' OR JSON_VALUE(nested, '$.nesteder.string1') = '1000'
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 10 5000000 false avgt 5 11.482 ± 0.495 ms/op
SqlNestedDataBenchmark.querySql 10 5000000 force avgt 5 11.549 ± 0.303 ms/op
SqlNestedDataBenchmark.querySql 11 5000000 false avgt 5 11.695 ± 0.293 ms/op
SqlNestedDataBenchmark.querySql 11 5000000 force avgt 5 11.931 ± 0.338 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 10 5000000 false avgt 5 11.427 ± 0.480 ms/op
SqlNestedDataBenchmark.querySql 10 5000000 force avgt 5 11.545 ± 0.431 ms/op
SqlNestedDataBenchmark.querySql 11 5000000 false avgt 5 11.650 ± 0.520 ms/op
SqlNestedDataBenchmark.querySql 11 5000000 force avgt 5 11.732 ± 0.406 ms/op
SELECT SUM(long1) FROM foo WHERE long2 = 10000 OR long2 = 1000
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) = 10000 OR JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) = 1000
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 12 5000000 false avgt 5 78.895 ± 2.158 ms/op
SqlNestedDataBenchmark.querySql 12 5000000 force avgt 5 48.814 ± 0.874 ms/op
SqlNestedDataBenchmark.querySql 13 5000000 false avgt 5 1.297 ± 0.008 ms/op
SqlNestedDataBenchmark.querySql 13 5000000 force avgt 5 1.277 ± 0.011 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 12 5000000 false avgt 5 77.838 ± 3.391 ms/op
SqlNestedDataBenchmark.querySql 12 5000000 force avgt 5 50.000 ± 1.199 ms/op
SqlNestedDataBenchmark.querySql 13 5000000 false avgt 5 1.096 ± 0.017 ms/op
SqlNestedDataBenchmark.querySql 13 5000000 force avgt 5 1.105 ± 0.023 ms/op
SELECT SUM(long1) FROM foo WHERE double3 < 10000.0 AND double3 > 1000.0
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) < 10000.0 AND JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) > 1000.0
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 14 5000000 false avgt 5 92.982 ± 1.473 ms/op
SqlNestedDataBenchmark.querySql 14 5000000 force avgt 5 54.729 ± 0.429 ms/op
SqlNestedDataBenchmark.querySql 15 5000000 false avgt 5 580.472 ± 28.064 ms/op
SqlNestedDataBenchmark.querySql 15 5000000 force avgt 5 561.494 ± 54.096 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 14 5000000 false avgt 5 93.883 ± 4.995 ms/op
SqlNestedDataBenchmark.querySql 14 5000000 force avgt 5 52.985 ± 1.123 ms/op
SqlNestedDataBenchmark.querySql 15 5000000 false avgt 5 228.775 ± 3.131 ms/op
SqlNestedDataBenchmark.querySql 15 5000000 force avgt 5 216.295 ± 2.309 ms/op
SELECT long1, SUM(double3) FROM foo WHERE string1 = '10000' OR string1 = '1000' GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.long1' RETURNING BIGINT), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.string1') = '10000' OR JSON_VALUE(nested, '$.nesteder.string1') = '1000' GROUP BY 1 ORDER BY 2
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 16 5000000 false avgt 5 129.760 ± 9.953 ms/op
SqlNestedDataBenchmark.querySql 16 5000000 force avgt 5 133.015 ± 20.961 ms/op
SqlNestedDataBenchmark.querySql 17 5000000 false avgt 5 142.197 ± 8.773 ms/op
SqlNestedDataBenchmark.querySql 17 5000000 force avgt 5 132.048 ± 15.546 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 16 5000000 false avgt 5 125.571 ± 3.131 ms/op
SqlNestedDataBenchmark.querySql 16 5000000 force avgt 5 125.625 ± 4.528 ms/op
SqlNestedDataBenchmark.querySql 17 5000000 false avgt 5 125.689 ± 2.543 ms/op
SqlNestedDataBenchmark.querySql 17 5000000 force avgt 5 126.233 ± 4.543 ms/op
SELECT string1, SUM(double3) FROM foo WHERE long2 < 10000 AND long2 > 1000 GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) < 10000 AND JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) > 1000 GROUP BY 1 ORDER BY 2
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 18 5000000 false avgt 5 161.445 ± 7.024 ms/op
SqlNestedDataBenchmark.querySql 18 5000000 force avgt 5 138.212 ± 19.673 ms/op
SqlNestedDataBenchmark.querySql 19 5000000 false avgt 5 123.486 ± 5.029 ms/op
SqlNestedDataBenchmark.querySql 19 5000000 force avgt 5 120.079 ± 6.822 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 18 5000000 false avgt 5 149.483 ± 4.772 ms/op
SqlNestedDataBenchmark.querySql 18 5000000 force avgt 5 128.978 ± 4.393 ms/op
SqlNestedDataBenchmark.querySql 19 5000000 false avgt 5 114.389 ± 4.373 ms/op
SqlNestedDataBenchmark.querySql 19 5000000 force avgt 5 114.224 ± 3.520 ms/op
SELECT string1, SUM(double3) FROM foo WHERE double3 < 10000.0 AND double3 > 1000.0 GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) < 10000.0 AND JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) > 1000.0 GROUP BY 1 ORDER BY 2
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 20 5000000 false avgt 5 280.393 ± 15.369 ms/op
SqlNestedDataBenchmark.querySql 20 5000000 force avgt 5 174.545 ± 2.702 ms/op
SqlNestedDataBenchmark.querySql 21 5000000 false avgt 5 802.647 ± 32.078 ms/op
SqlNestedDataBenchmark.querySql 21 5000000 force avgt 5 591.274 ± 16.460 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 20 5000000 false avgt 5 264.055 ± 4.947 ms/op
SqlNestedDataBenchmark.querySql 20 5000000 force avgt 5 172.410 ± 3.586 ms/op
SqlNestedDataBenchmark.querySql 21 5000000 false avgt 5 454.973 ± 5.799 ms/op
SqlNestedDataBenchmark.querySql 21 5000000 force avgt 5 359.288 ± 5.179 ms/op
SELECT long2 FROM foo WHERE long2 IN (1, 19, 21, 23, 25, 26, 46),
SELECT JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) IN (1, 19, 21, 23, 25, 26, 46),
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 22 5000000 false avgt 5 273.464 ± 15.731 ms/op
SqlNestedDataBenchmark.querySql 22 5000000 force avgt 5 272.270 ± 20.511 ms/op
SqlNestedDataBenchmark.querySql 23 5000000 false avgt 5 174.960 ± 1.923 ms/op
SqlNestedDataBenchmark.querySql 23 5000000 force avgt 5 177.920 ± 4.095 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 22 5000000 false avgt 5 283.230 ± 6.222 ms/op
SqlNestedDataBenchmark.querySql 22 5000000 force avgt 5 282.467 ± 5.709 ms/op
SqlNestedDataBenchmark.querySql 23 5000000 false avgt 5 177.047 ± 4.990 ms/op
SqlNestedDataBenchmark.querySql 23 5000000 force avgt 5 172.208 ± 1.031 ms/op
SELECT long2 FROM foo WHERE long2 IN (1, 19, 21, 23, 25, 26, 46) GROUP BY 1",
SELECT JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) IN (1, 19, 21, 23, 25, 26, 46) GROUP BY 1
old:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 24 5000000 false avgt 5 318.280 ± 7.544 ms/op
SqlNestedDataBenchmark.querySql 24 5000000 force avgt 5 210.866 ± 14.684 ms/op
SqlNestedDataBenchmark.querySql 25 5000000 false avgt 5 215.200 ± 2.366 ms/op
SqlNestedDataBenchmark.querySql 25 5000000 force avgt 5 152.399 ± 22.695 ms/op
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 24 5000000 false avgt 5 308.668 ± 7.274 ms/op
SqlNestedDataBenchmark.querySql 24 5000000 force avgt 5 199.587 ± 3.162 ms/op
SqlNestedDataBenchmark.querySql 25 5000000 false avgt 5 212.399 ± 4.406 ms/op
SqlNestedDataBenchmark.querySql 25 5000000 force avgt 5 149.650 ± 3.489 ms/op
SELECT SUM(long1) FROM foo WHERE double3 < 1005.0 AND double3 > 1000.0
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) < 1005.0 AND JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) > 1000.0
old:
(not previously measured)
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 26 5000000 false avgt 5 74.495 ± 1.506 ms/op
SqlNestedDataBenchmark.querySql 26 5000000 force avgt 5 48.270 ± 0.806 ms/op
SqlNestedDataBenchmark.querySql 27 5000000 false avgt 5 12.997 ± 0.509 ms/op
SqlNestedDataBenchmark.querySql 27 5000000 force avgt 5 13.094 ± 0.553 ms/op
SELECT SUM(long1) FROM foo WHERE double3 < 2000.0 AND double3 > 1000.0
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) < 2000.0 AND JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) > 1000.0
old:
(not previously measured)
new:
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 28 5000000 false avgt 5 79.335 ± 2.919 ms/op
SqlNestedDataBenchmark.querySql 28 5000000 force avgt 5 51.953 ± 1.056 ms/op
SqlNestedDataBenchmark.querySql 29 5000000 false avgt 5 40.987 ± 0.710 ms/op
SqlNestedDataBenchmark.querySql 29 5000000 force avgt 5 40.654 ± 0.713 ms/op
```
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2709157142
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2375491770
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2781755864
This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] imply-cheddar commented on a diff in pull request #12846: specialized FixedIndexed implementations for java value types
Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12846:
URL: https://github.com/apache/druid/pull/12846#discussion_r965528921
##########
processing/src/main/java/org/apache/druid/segment/data/FixedIndexedDoubles.java:
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import it.unimi.dsi.fastutil.doubles.DoubleComparator;
+import it.unimi.dsi.fastutil.doubles.DoubleComparators;
+import it.unimi.dsi.fastutil.doubles.DoubleIterator;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Iterator;
+
+/**
+ * Specialized implementation for {@link FixedIndexed<Double>} which does not contain any null values, allowing it to
+ * deal in java double value types instead of {@link Double} objects, and utilize specialized {@link ByteBuffer} methods
+ * to more efficiently read data.
+ */
+public final class FixedIndexedDoubles implements Indexed<Double>, HotLoopCallee
+{
+ public static FixedIndexedDoubles read(ByteBuffer bb, ByteOrder byteOrder)
+ {
+ final ByteBuffer buffer = bb.asReadOnlyBuffer().order(byteOrder);
+ final byte version = buffer.get();
+ Preconditions.checkState(version == 0, "Unknown version [%s]", version);
+ final byte flags = buffer.get();
+ final boolean hasNull = (flags & NullHandling.IS_NULL_BYTE) == NullHandling.IS_NULL_BYTE ? true : false;
+ final boolean isSorted = (flags & FixedIndexed.IS_SORTED_MASK) == FixedIndexed.IS_SORTED_MASK ? true : false;
+ Preconditions.checkState(!hasNull, "Cannot use FixedIndexedInts for FixedIndex with null values");
+ Preconditions.checkState(!(hasNull && !isSorted), "cannot have null values if not sorted");
Review Comment:
I'm pretty sure that this precondition will always be true if the precondition of `!hasNull` from the line above is true.
##########
processing/src/main/java/org/apache/druid/segment/data/FixedIndexedInts.java:
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import it.unimi.dsi.fastutil.ints.IntComparator;
+import it.unimi.dsi.fastutil.ints.IntComparators;
+import it.unimi.dsi.fastutil.ints.IntIterator;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Iterator;
+
+/**
+ * Specialized implementation for {@link FixedIndexed<Integer>} which does not contain any null values, allowing it to
+ * deal in java int value types instead of {@link Integer} objects, and utilize specialized {@link ByteBuffer} methods
+ * to more efficiently read data.
+ */
+public final class FixedIndexedInts implements Indexed<Integer>, HotLoopCallee
+{
+ public static FixedIndexedInts read(ByteBuffer bb, ByteOrder byteOrder)
+ {
+ final ByteBuffer buffer = bb.asReadOnlyBuffer().order(byteOrder);
+ final byte version = buffer.get();
+ Preconditions.checkState(version == 0, "Unknown version [%s]", version);
+ final byte flags = buffer.get();
+ final boolean hasNull = (flags & NullHandling.IS_NULL_BYTE) == NullHandling.IS_NULL_BYTE ? true : false;
+ final boolean isSorted = (flags & FixedIndexed.IS_SORTED_MASK) == FixedIndexed.IS_SORTED_MASK ? true : false;
+ Preconditions.checkState(!hasNull, "Cannot use FixedIndexedInts for FixedIndex with null values");
+ Preconditions.checkState(!(hasNull && !isSorted), "cannot have null values if not sorted");
+ final int size = buffer.getInt() + (hasNull ? 1 : 0);
+ final int valuesOffset = buffer.position();
+ final FixedIndexedInts fixedIndexed = new FixedIndexedInts(
+ buffer,
+ isSorted,
+ size,
+ valuesOffset
+ );
+ bb.position(buffer.position() + (Integer.BYTES * size));
+ return fixedIndexed;
+ }
+
+ private final ByteBuffer buffer;
+ private final int size;
+ private final int valuesOffset;
+ private final boolean isSorted;
+ private final IntComparator comparator;
+
+ private FixedIndexedInts(
+ ByteBuffer buffer,
+ boolean isSorted,
+ int size,
+ int valuesOffset
+ )
+ {
+ this.buffer = buffer;
+ this.size = size;
+ this.valuesOffset = valuesOffset;
+ this.isSorted = isSorted;
+ this.comparator = IntComparators.NATURAL_COMPARATOR;
+ }
+
+ @Override
+ public int size()
+ {
+ return size;
+ }
+
+ @Nullable
+ @Override
+ public Integer get(int index)
+ {
+ return getInt(index);
+ }
+
+ @Override
+ public int indexOf(@Nullable Integer value)
+ {
+ if (value == null) {
+ return -1;
+ }
+ return indexOf(value.intValue());
+ }
+
+ public int getInt(int index)
+ {
+ return buffer.getInt(valuesOffset + (index * Integer.BYTES));
+ }
+
+ public int indexOf(int value)
+ {
+ if (!isSorted) {
+ throw new UnsupportedOperationException("Reverse lookup not allowed.");
+ }
+ int minIndex = 0;
+ int maxIndex = size - 1;
+ while (minIndex <= maxIndex) {
+ int currIndex = (minIndex + maxIndex) >>> 1;
+
+ int currValue = getInt(currIndex);
+ int comparison = comparator.compare(currValue, value);
+ if (comparison == 0) {
+ return currIndex;
+ }
+
+ if (comparison < 0) {
+ minIndex = currIndex + 1;
+ } else {
+ maxIndex = currIndex - 1;
+ }
+ }
+
+ return -(minIndex + 1);
+ }
+
+ public IntIterator intIterator()
+ {
+ final ByteBuffer copy = buffer.asReadOnlyBuffer().order(buffer.order());
+ copy.position(valuesOffset);
+ copy.limit(valuesOffset + (size * Integer.BYTES));
+ return new IntIterator()
+ {
+ @Override
+ public int nextInt()
+ {
+ return copy.getInt();
+ }
+
+ @Override
+ public boolean hasNext()
+ {
+ return copy.hasRemaining();
+ }
+ };
+ }
+
+ @Override
+ public Iterator<Integer> iterator()
Review Comment:
It looks like `IntIterator` already implements `Iterator<Integer>` so you don't need to wrap it again.
##########
processing/src/main/java/org/apache/druid/segment/data/FixedIndexedDoubleWriter.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import it.unimi.dsi.fastutil.doubles.DoubleIterator;
+import org.apache.druid.io.Channels;
+import org.apache.druid.java.util.common.io.smoosh.FileSmoosher;
+import org.apache.druid.segment.serde.Serializer;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+import org.apache.druid.segment.writeout.WriteOutBytes;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.WritableByteChannel;
+
+/**
+ * Specialized version of {@link FixedIndexedWriter} for writing double value types, with no support for null values,
+ * and no verification that data is actually sorted. The resulting data can be read into either
+ * {@link FixedIndexedDoubles} or a {@link FixedIndexed<Double>}, since the format is identical.
+ *
+ * Callers should be certain that the data written is in fact sorted if specifying it as such. If null values need
+ * to be stored then the generic {@link FixedIndexedWriter} should be used instead.
+ */
+public class FixedIndexedDoubleWriter implements Serializer
+{
+ private static final int PAGE_SIZE = 4096;
+ private final SegmentWriteOutMedium segmentWriteOutMedium;
+ private final ByteBuffer scratch;
+ private int numWritten;
+ @Nullable
+ private WriteOutBytes valuesOut = null;
+
+ private final boolean isSorted;
+
+ public FixedIndexedDoubleWriter(SegmentWriteOutMedium segmentWriteOutMedium, boolean sorted)
+ {
+ this.segmentWriteOutMedium = segmentWriteOutMedium;
+ // this is a matter of faith, nothing checks
+ this.isSorted = sorted;
Review Comment:
Do we need to know if it's sorted at this level? If we are being told that it is supposed to be sorted, then I think that we should verify that it's sorted. We can do that relatively easily by just having the most-recent value, checking that the next one is greater and then storing that.
Or, if we are trusting something external from us to know that it is already sorted, then we shouldn't even bother asking it to tell us if we are sorted or not and instead just expect the external code to do the right thing.
##########
processing/src/main/java/org/apache/druid/segment/nested/NestedFieldLiteralColumnIndexSupplier.java:
##########
@@ -622,7 +630,7 @@ public Iterable<ImmutableBitmap> getBitmapIterable()
final DruidLongPredicate longPredicate = matcherFactory.makeLongPredicate();
// in the future, this could use an int iterator
- final Iterator<Integer> iterator = dictionary.iterator();
+ final Iterator<Integer> iterator = dictionary.intIterator();
Review Comment:
you switched to an intIterator, but are still using an `Iterator<Integer>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2106067571
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12846: specialized FixedIndexed implementations for java value types
URL: https://github.com/apache/druid/pull/12846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12846: specialized FixedIndexed implementations for java value types
URL: https://github.com/apache/druid/pull/12846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2433863045
This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2253669943
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-1880241495
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-1989679984
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] specialized FixedIndexed implementations for java value types (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12846:
URL: https://github.com/apache/druid/pull/12846#issuecomment-2560463417
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org