You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "Aitozi (via GitHub)" <gi...@apache.org> on 2023/08/10 12:11:47 UTC
[GitHub] [orc] Aitozi opened a new issue, #1576: How to use the bloomfilter in orc format
Aitozi opened a new issue, #1576:
URL: https://github.com/apache/orc/issues/1576
I want to test a value whether is in one orc file by the bloom filter, if in I want to load the file and cache the entire file for later lookup. However, it seems not have an interface like `BloomFilterReader` in parquet for using the internal bloom filter. Are there some good way to achieve this ?
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to use the bloomfilter in orc format [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #1576:
URL: https://github.com/apache/orc/issues/1576#issuecomment-1815150574
Hi, @Aitozi . Did you solve your problem?
I guess the original question was an lookup API which currently we don't provide.
Like Apache Iceberg, Apache Spark also provides only a high-level usage documentation.
- https://spark.apache.org/docs/latest/sql-data-sources-orc.html#bloom-filters
```SQL
CREATE TABLE users_with_options (
name STRING,
favorite_color STRING,
favorite_numbers array<integer>
)
USING ORC
OPTIONS (
orc.bloom.filter.columns 'favorite_color',
orc.dictionary.key.threshold '1.0',
orc.column.encoding.direct 'name'
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to use the bloomfilter in orc format [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed issue #1576: How to use the bloomfilter in orc format
URL: https://github.com/apache/orc/issues/1576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] Aitozi commented on issue #1576: How to use the bloomfilter in orc format
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on issue #1576:
URL: https://github.com/apache/orc/issues/1576#issuecomment-1674146386
@dongjoon-hyun Thanks for your quick response. I'm exploring to leverage the BloomFilter to speed up the [`lookup`](https://paimon.apache.org/docs/master/concepts/primary-key-table/#lookup) in Paimon compaction process.
i want to judge the key in or not in a orc file by the min/max statistics and bloomfilter. If in, I will load the orc file to build a file cache for further lookup. However, bloomfilter in orc format seems not accessible outside the reader, Can you give me some suggestion here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] wgtmac commented on issue #1576: How to use the bloomfilter in orc format
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #1576:
URL: https://github.com/apache/orc/issues/1576#issuecomment-1675983740
> @dongjoon-hyun Thanks for your quick response. I'm exploring to leverage the BloomFilter to speed up the [`lookup`](https://paimon.apache.org/docs/master/concepts/primary-key-table/#lookup) in Paimon compaction process.
>
> i want to judge the key in or not in a orc file by the min/max statistics and bloomfilter. If in, I will load the orc file to build a file cache for further lookup. However, bloomfilter in orc format seems not accessible outside the reader, Can you give me some suggestion here ?
Apache Iceberg has supported this. Please check this for more information: https://github.com/apache/iceberg/issues/5218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on issue #1576: How to use the bloomfilter in orc format
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #1576:
URL: https://github.com/apache/orc/issues/1576#issuecomment-1673582804
Thank you for filing issues, @Aitozi . May I ask what is your use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to use the bloomfilter in orc format [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on issue #1576:
URL: https://github.com/apache/orc/issues/1576#issuecomment-1818186732
Let me close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org