You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Anthony Pessy (JIRA)" <ji...@apache.org> on 2019/06/18 12:08:00 UTC
[jira] [Comment Edited] (PARQUET-1602) PageIndex not working as
suggested ?
[ https://issues.apache.org/jira/browse/PARQUET-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866549#comment-16866549 ]
Anthony Pessy edited comment on PARQUET-1602 at 6/18/19 12:07 PM:
------------------------------------------------------------------
For the following output (sorry for approximative indentation, lost during copy/paste):
{noformat}
row group 0:
column index for column url:
Boudary order: ASCENDING
null count min max
page-0 0 http://materiais.(...)delos-de-curriculo https://api.quero(...)954874/toogle_like
page-1 0 https://api.quero(...)880/toogle_dislike https://api.quero(...)ior-online/encceja
page-2 0 https://api.quero(...)erior-online/todos https://api.quero(...)nte-em-saude/todos
offset index for column url:
offset compressed size first row index
page-0 4 224274 0
page-1 224278 100168 20000
page-2 324446 67778 40000
column index for column content:
NONE
offset index for column content:
offset compressed size first row index
page-0 392224 504412 0
page-1 896636 784246 125
page-2 1680882 641212 200
[...truncated...]
page-596 256651848 183162 53100{noformat}
When looking for the url matching the 'max' of `page-0` ( `[https://api.quero(...)954874/toogle_like|https://api.quero%28...%29954874/toogle_like]`) I have in total 213 pages loaded (1 for url and 212 for content) (~230MB of data transferred).
Offsets for content:
{code:java}
page-212 98231687 206970 26750
page-213 98438657 501408 26850
{code}
It would imply that it loaded every page of content until until the matching row.
On a separate note when the tool says `[https://api.quero(...)954874/toogle_like|https://api.quero%28...%29954874/toogle_like]`, do you now if it's a display thing or actually truncated like that ? I set the index max length 512 when creating the file.
was (Author: panthony):
For the following output (sorry for approximative indentation, lost during copy/paste):
{noformat}
row group 0:
column index for column url:
Boudary order: ASCENDING
null count min max
page-0 0 http://materiais.(...)delos-de-curriculo https://api.quero(...)954874/toogle_like
page-1 0 https://api.quero(...)880/toogle_dislike https://api.quero(...)ior-online/encceja
page-2 0 https://api.quero(...)erior-online/todos https://api.quero(...)nte-em-saude/todos
offset index for column url:
offset compressed size first row index
page-0 4 224274 0
page-1 224278 100168 20000
page-2 324446 67778 40000
column index for column content:
NONE
offset index for column content:
offset compressed size first row index
page-0 392224 504412 0
page-1 896636 784246 125
page-2 1680882 641212 200
[...truncated...]
page-596 256651848 183162 53100{noformat}
When looking for the url matching the 'max' of `page-0` ( `[https://api.quero(...)954874/toogle_like|https://api.quero%28...%29954874/toogle_like]`) I have in total 213 pages loaded (1 for url and 212 for content) (~230MB of data transferred).
Offsets for content:
{code:java}
page-212 98231687 206970 26750
page-213 98438657 501408 26850
{code}
It would imply that it loaded every page of content until until the matching row.
On a separate note when the tool says `[https://api.quero(...)954874/toogle_like|https://api.quero%28...%29954874/toogle_like]`, do you now if it's a display thing or actually truncated like that ? I set the index max length 512 when creating the file.
> PageIndex not working as suggested ?
> ------------------------------------
>
> Key: PARQUET-1602
> URL: https://issues.apache.org/jira/browse/PARQUET-1602
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.11.0
> Reporter: Anthony Pessy
> Priority: Major
>
> I have a schema such as:
>
> ```
> {{schema message pages {}}
> {{ required binary url (STRING);}}
> {{ optional binary content (STRING);}}
> {{}}}
> ```
>
> Where `url` is unique and ordered, the file is created in such a way that I have ~600 pages of `content` for 1 page of `url`.
>
> From [https://github.com/apache/parquet-format/blob/master/PageIndex.md] I saw:
> > A single-row lookup in a rowgroup based on the sort column of that rowgroup will only read one data page per retrieved column.
>
>
> I was expecting `ParquetReader` to find the matching row thanks to the `FilterPredicate` on `url`, decoding only this column, then, using `offset index`, directly seek to the appropriate page for `content` and decode it.
>
>
> Instead, what I'm seeing, is that the reader fully reads & decode the ~600 pages of content (until it actually find the url).
>
> Is there something I misunderstood or some step to ensure to make the reader only consume the necessary pages?
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)