You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "alippai (via GitHub)" <gi...@apache.org> on 2023/11/21 04:26:39 UTC
[I] Parquet dictionary filter pushdown slow [arrow]
alippai opened a new issue, #38818:
URL: https://github.com/apache/arrow/issues/38818
### Describe the enhancement requested
I have a sorted column of type `pa.utf8()`.
If I write it directly and set the parquet `use_dictionary`, the `read_table(..., filters=[('column', '=', 'value')])` is fast.
If I convert it to dictionary first using `pc.dictionary_encode()` and save it similarly, the same filter is 10-20x slower.
Checking the file with parquet-cli the files the metadata and the rowgroups are almost identical (eg. stats). However when checking the pages they differ wildly, I guess that's the reason for the speed difference. Each dict page has every entry when saving the dictionary, each dict page has a dozen entries only when saving the raw strings.
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825177638
`dict.parquet` is 20x larger than `nodict.parquet`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825196595
it depend on the impl of filter, I may dive into it later. Currently I found:
nodict:
```
Write: ["0", "0", ...]
The column will build a dict.
dict: ["0"]
indices: [0, 0, ... 0]
min: "0"
max: "0"
```
dict:
```
Write: dict: ["0", "1", ..."9999"], indices: [0, ..., 0]
The column will reuse dict.
dict: ["0", "1", ..."9999"]
indices: [0, ..., 0]
min: "0"
max: "0"
```
I think the filter might try prune the RowGroup, the min-max is right, so they'll prune the same data. But when reading, the dict will use more time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825242918
Ah, you are right. That would be a weird hack.
On the other hand reading the spec further it looks like the statistics are storing the values. Only one row group should be read. I can try strace tomorrow, but I have a feeling all row groups are read. I would be surprised if reading a dictionary with 10k values would take 0.5s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825217993
> 2.) same dictionary pages are not reused when writing
Hmmm Parquet RowGroup is a unit for reading, so here its just a RowGroup-level dictionary. During reading, it don't knowing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825199140
It's no so easy to detect it in writer 🤔 It's so easy to found some counter-case in senerio like this...
You can just use non-dict write as a workaround
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-3921972630
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825216069
Sounds like we are facing three problems:
1.) unused dict values are not pruned in dict pages
2.) same dictionary pages are not reused when writing
3.) when reading the same dictionary page it’s not cached
I wonder why didn’t I see this with integer-integer dictionary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825198042
Ideally if the dictionary page is below a threshold it should be reused and not pruned. I think the parquet format allows reusing a dict page for the full column or is this a missing feature? Writing the full multiple times might not be necessary even without pruning, writing many small dict pages with a few values sounds inefficient as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1821916940
@mapleFU as you are working in dict encoding, this might be interesting for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825184360
Oh finally I got the answer @alippai . This is a logical problem in arrow write dictionary. You can first try to convert it to utf8 during write.
It doesn't because filter pushdown is slow. It's because when writing a dictionary to parquet, the dictionary is directly write to parquet without pruning unused dict items.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow and large files are created [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1825189893
So reads all the dict pages (unnecessarily) for the search string and only after that it evaluates the statistics? And it has to read it because other writers or other data types prune the entries first.
Could pyarrow use the arrow schema instead of the dict pages if present?
But ideally pyarrow should prune the values when writing an utf8 dict, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet dictionary filter pushdown slow [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #38818:
URL: https://github.com/apache/arrow/issues/38818#issuecomment-1821811099
It happens with strings only, numbers work as expected.
I little bit ugly example:
```python
pq.write_table(
pa.Table.from_arrays(
[
pc.dictionary_encode(
pa.array(np.sort(pa.array(list(range(10000)) * 1000)), type=pa.int16())
)
],
None,
pa.schema([pa.field("a", pa.dictionary(pa.int16(), pa.utf8()), False)]),
),
"dict.parquet",
data_page_version="2.0",
row_group_size=10000,
use_dictionary=["a"],
compression="zstd",
dictionary_pagesize_limit=100 * 1024 * 1024,
)
```
vs
```python
pq.write_table(
pa.Table.from_arrays(
[pa.array(np.sort(pa.array(list(range(10000)) * 1000)), type=pa.int16()).cast(pa.utf8())],
None,
pa.schema([pa.field("a", pa.utf8(), False)]),
),
"nodict.parquet",
data_page_version="2.0",
row_group_size=10000,
use_dictionary=["a"],
compression="zstd",
dictionary_pagesize_limit=100*1024*1024
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org