[jira] [Commented] (PARQUET-1602) PageIndex not working as suggested ?

["Gabor Szadovszky (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866498#comment-16866498 ] 

Gabor Szadovszky commented on PARQUET-1602:
-------------------------------------------

You can check the page and offset indexes in the parquet file by using `parquet-tools column-index` or `parquet-cli column-index`. (exec with `--help` for details)
Could you check if the value you're searching for is really between the min/max values of the 1 page only?

> PageIndex not working as suggested ?
> ------------------------------------
>
>                 Key: PARQUET-1602
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1602
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>    Affects Versions: 1.11.0
>            Reporter: Anthony Pessy
>            Priority: Major
>
> I have a schema such as:
>  
> ``` 
> {{schema message pages {}}
> {{  required binary url (STRING);}}
> {{  optional binary content (STRING);}}
> {{}}}
> ``` 
>  
> Where `url` is unique and ordered, the file is created in such a way that I have ~600 pages of `content` for 1 page of `url`.
>  
> From [https://github.com/apache/parquet-format/blob/master/PageIndex.md] I saw:
> > A single-row lookup in a rowgroup based on the sort column of that rowgroup will only read one data page per retrieved column.
>  
>  
> I was expecting `ParquetReader`  to find the matching row thanks to the `FilterPredicate`  on `url`, decoding only this column, then, using `offset index`, directly seek to the appropriate page for `content` and decode it.
>  
>  
> Instead, what I'm seeing, is that the reader fully reads & decode the ~600 pages of content (until it actually find the url).
>  
> Is there something I misunderstood or some step to ensure to make the reader only consume the necessary pages?
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)