You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Anthony Pessy (JIRA)" <ji...@apache.org> on 2019/06/18 14:27:00 UTC
[jira] [Resolved] (PARQUET-1602) PageIndex not working as suggested
?
[ https://issues.apache.org/jira/browse/PARQUET-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anthony Pessy resolved PARQUET-1602.
------------------------------------
Resolution: Not A Problem
> PageIndex not working as suggested ?
> ------------------------------------
>
> Key: PARQUET-1602
> URL: https://issues.apache.org/jira/browse/PARQUET-1602
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.11.0
> Reporter: Anthony Pessy
> Priority: Major
>
> I have a schema such as:
> {code:java}
> schema message pages {
> required binary url (STRING);
> optional binary content (STRING);
> }
> {code}
> Where `url` is unique and ordered, the file is created in such a way that I have ~600 pages of `content` for 1 page of `url`.
>
> From [https://github.com/apache/parquet-format/blob/master/PageIndex.md] I saw:
> {noformat}
> A single-row lookup in a rowgroup based on the sort column of that rowgroup will only read one data page per retrieved column.{noformat}
> I was expecting `ParquetReader` to find the matching row thanks to the `FilterPredicate` on `url`, decoding only this column, then, using `offset index`, directly seek to the appropriate page for `content` and decode it.
> Instead, what I'm seeing, is that the reader fully reads & decode the ~600 pages of content (until it actually find the url).
> Is there something I misunderstood or some step to ensure to make the reader only consume the necessary pages?
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)