You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/12/30 12:31:04 UTC
[I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
alamb opened a new issue, #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685
> How do you know the bloom filter isn't being used? Is there a reproducer (a parquet file) you can share?
>
> It appears that there is no good way to know if the bloom filter code is working via logging or metrics 🤔
>
> https://github.com/apache/arrow-datafusion/blob/f39c040ace0b34b0775827907aa01d6bb71cbb14/datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs#L111-L168
I conducted a test locally by writing 200GB of data. When using a Bloom filter for queries, I observed that the query only takes 0.1 seconds, whereas without using the Bloom filter, the query takes 1 second. If a query takes 1 second, I can infer that it is not using the Bloom filter because using the Bloom filter should yield results within 0.1 seconds.
_Originally posted by @domyway in https://github.com/apache/arrow-datafusion/issues/8436#issuecomment-1871854464_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1872523407
I tried to add more detailed metric for bloomfilters. Codes here https://github.com/apache/arrow-datafusion/compare/main...my-vegetable-has-exploded:arrow-datafusion:metric-sbbf?expand=1, it works well on unit tests. But when I build datafusion-cli, it fails to execute `EXPLAIN` command.
```
❯ EXPLAIN ANALYZE SELECT * FROM taxi WHERE (taxi."String" IN ('a'));
Internal error: Optimization not supported for ANALYZE.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Regression: bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1884107159
Hi @domyway, you can check whether bloom filter works by `row_groups_pruned_bloom_filter` metric now.
In my environment, bloom filter works.
```
❯ CREATE EXTERNAL TABLE taxi
STORED AS PARQUET
LOCATION '/home/deepin/rust/arrow-datafusion/parquet-testing/data/data_index_bloom_encoding_stats.parquet'
;
0 rows in set. Query took 0.002 seconds.
❯ SET datafusion.execution.parquet.bloom_filter_enabled to true;
0 rows in set. Query took 0.001 seconds.
❯ EXPLAIN ANALYZE SELECT * FROM taxi WHERE (taxi."String" IN ('bb', 'bbc', 'bba', 'bbd', 'bbg', 'bbf', 'bbn', 'nnfa', 'bbnfd', 'bbx', 'bbxda', 'badfas', 'afd', 'adfas', 'adfa', 'asdfer', 'sefarj', 'erseioio', 'uioosdf', '0ba24'));
....row_groups_pruned_bloom_filter=1, .....
```
Thank you for finding it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1874203912
> > it works well on unit tests. But when I build datafusion-cli, it fails to execute `EXPLAIN` command.
>
> I filed #8690 to track
The issue has been fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1872519715
I think the next step to proceed here would be to get some sort of reproducer so we can debug further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8685: Report that bloom filters are not being used in Parquet queries
URL: https://github.com/apache/arrow-datafusion/issues/8685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1875130287
good catch @domyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1875262769
For anyone following along, the fix is https://github.com/apache/arrow-datafusion/pull/8732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Report that bloom filters are not being used in Parquet queries [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8685:
URL: https://github.com/apache/arrow-datafusion/issues/8685#issuecomment-1872553362
> it works well on unit tests. But when I build datafusion-cli, it fails to execute `EXPLAIN` command.
I filed https://github.com/apache/arrow-datafusion/issues/8690 to track
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org