[jira] [Created] (PARQUET-1602) PageIndex not working as suggested ?

["Anthony Pessy (JIRA)" <ji...@apache.org>]

Anthony Pessy created PARQUET-1602:
--------------------------------------

             Summary: PageIndex not working as suggested ?
                 Key: PARQUET-1602
                 URL: https://issues.apache.org/jira/browse/PARQUET-1602
             Project: Parquet
          Issue Type: Improvement
          Components: parquet-mr
    Affects Versions: 1.11.0
            Reporter: Anthony Pessy


I have a schema such as:

 

``` 

{{schema message pages {}}

{{  required binary url (STRING);}}

{{  optional binary content (STRING);}}

{{}}}

``` 

 

Where `url` is unique and ordered, the file is created in such a way that I have ~600 pages of `content` for 1 page of `url`.

 

From [https://github.com/apache/parquet-format/blob/master/PageIndex.md] I saw:

> A single-row lookup in a rowgroup based on the sort column of that rowgroup will only read one data page per retrieved column.

 

 

I was expecting `ParquetReader`  to find the matching row thanks to the `FilterPredicate`  on `url`, decoding only this column, then, using `offset index`, directly seek to the appropriate page for `content` and decode it.

 

 

Instead, what I'm seeing, is that the reader fully reads & decode the ~600 pages of content (until it actually find the url).

 

Is there something I misunderstood or some step to ensure to make the reader only consume the necessary pages?

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)