You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/11/05 15:18:02 UTC
[GitHub] [incubator-druid] jnaous commented on issue #8822: optimize numeric
column null value checking for low filter selectivity (more rows)
jnaous commented on issue #8822: optimize numeric column null value checking for low filter selectivity (more rows)
URL: https://github.com/apache/incubator-druid/pull/8822#issuecomment-549866189
Thanks Clint! These /are/ really good benchmarks. I'm curious what
percentage of the entire cost for processing a row a null check would be so
we can have a good idea of what % overhead we're talking about.
On Tue, Nov 5, 2019 at 12:36 AM Clint Wylie <no...@github.com>
wrote:
> To follow-up on the PR description, I let the concise benchmarks finish
> where nearly all of the rows are selected:
>
> Benchmark (bitmapType) (filterMatch) (nullDensity) (numRows) Mode Cnt Score Error Units
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.99999 0 500000 avgt 2 2438.318 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.99999 0.1 500000 avgt 2 261150067.426 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.99999 0.25 500000 avgt 2 430133586.339 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.99999 0.5 500000 avgt 2 623322588.940 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.99999 0.75 500000 avgt 2 449875260.568 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.99999 0.99 500000 avgt 2 29015358.505 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.99999 0 500000 avgt 2 2316.307 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.99999 0.1 500000 avgt 2 4158.871 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.99999 0.25 500000 avgt 2 6041.184 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.99999 0.5 500000 avgt 2 8397.707 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.99999 0.75 500000 avgt 2 6129.792 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.99999 0.99 500000 avgt 2 3511.961 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.99999 0 500000 avgt 2 2440.615 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.99999 0.1 500000 avgt 2 4431.640 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.99999 0.25 500000 avgt 2 6302.718 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.99999 0.5 500000 avgt 2 8911.671 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.99999 0.75 500000 avgt 2 6809.052 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.99999 0.99 500000 avgt 2 4995.936 us/op
>
> Using get with concise is just terribad. The results for 1% of rows is
> similarly pretty awful, though a couple orders of magnitude less bad:
>
> Benchmark (bitmapType) (filterMatch) (nullDensity) (numRows) Mode Cnt Score Error Units
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.01 0 500000 avgt 2 100.110 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.01 0.1 500000 avgt 2 3878501.478 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.01 0.25 500000 avgt 2 4853693.538 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.01 0.5 500000 avgt 2 8399206.128 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.01 0.75 500000 avgt 2 5169724.280 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.get concise 0.01 0.99 500000 avgt 2 440739.562 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.01 0 500000 avgt 2 69.756 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.01 0.1 500000 avgt 2 1813.610 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.01 0.25 500000 avgt 2 3364.983 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.01 0.5 500000 avgt 2 3745.194 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.01 0.75 500000 avgt 2 2702.778 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.iterator concise 0.01 0.99 500000 avgt 2 1874.390 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.01 0 500000 avgt 2 77.929 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.01 0.1 500000 avgt 2 1592.171 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.01 0.25 500000 avgt 2 2216.227 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.01 0.5 500000 avgt 2 3535.110 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.01 0.75 500000 avgt 2 2400.203 us/op
> NullHandlingBitmapGetVsIteratorBenchmark.peekableIterator concise 0.01 0.99 500000 avgt 2 436.322 us/op
>
> I did not plot these because it seemed to be rather pointless, concise
> seems to be *only* viable when using IntIterator or PeekableIntIterator.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-druid/pull/8822?email_source=notifications&email_token=AAPSYCQ2URIZF6VEJ3SSGADQSEWAJA5CNFSM4JI4LCNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDCATDY#issuecomment-549718415>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAPSYCV5IXHLEI6HE5RKM3DQSEWAJANCNFSM4JI4LCNA>
> .
>
--
Jad Naous
Imply | VP R&D
650-521-3425
jad.naous@imply.io
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org