You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "r2evans (via GitHub)" <gi...@apache.org> on 2024/01/17 20:56:25 UTC
[I] efficiently combine parquet files [arrow]
r2evans opened a new issue, #39671:
URL: https://github.com/apache/arrow/issues/39671
### Describe the enhancement requested
I recognize that appending to parquet files is not on the roadmap. Is it possible to do an efficient concatenation of two parquets with the output to a parquet file? While brute-force methods exist (read all of "A", read all of "B", and row-concatenate them however the language allows), it requires loading all data into memory. (I'm specifically targeting R, where it's perhaps more difficult to use the lower-level API.)
Part of the alternative to the "append" request (such as https://github.com/apache/arrow/issues/32708) is https://github.com/apache/arrow/issues/32708#issuecomment-1378120110:
> the pattern that Arrow enables is writing multiple files and then using open_dataset() to query them lazily
This works fine in concept, though as the count of files increases, eventually there is a tradeoff with performance. This penalty can be mitigated (e.g., `unify_schemas=FALSE`), but eventually there may be a time when there is the desire to reduce the number of files by combining them. The brute-force read of both _works_, but it would be very nice to have a simple function that takes 1+ input filenames and 1 output filename (previously non-existent) and as efficiently as possible concatenates the data (handling meta, of course). I'm guessing there would need to be assumptions/requirements with regards to the schemas between the files, perhaps a first guess would require "effectively identical" (where "effectively" might allow differences such as `numeric`/`integer` or similar), but I'd still be very happy with "perfectly identical".
I'm specifically targeting R in my usage, though I guess that other languages might also take advantage of this.
Thanks!
### Component(s)
Parquet, R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "char101 (via GitHub)" <gi...@apache.org>.
char101 commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-1902084793
Fastparquet also can append existing file by rewriting the footer
https://github.com/dask/fastparquet/blob/fb545a5d8147eb111eded0d5ac11eda03c574134/fastparquet/writer.py#L966-L984
Unfortunately fastparquest can't do delta encoding yet and I find that with time series data, delta encoding can reduce the compression by 30% more. It would be great if this can be added to pyarrow too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "r2evans (via GitHub)" <gi...@apache.org>.
r2evans commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-1901069004
That's an interesting utility, thank you for the pointer to it.
I had been thinking of capability within a particular language, perhaps something baked into `arrow.so` or similar. Frankly, I don't have `java` installed where this would be used, and I'm not eager to install it just for this utility.
Is it safe to infer that since it exists there in java, there is not an immediate desire to have a _compiled_ (with no JRE required) binary that does simple concatenation/rewrite?
Thank you again for the fast reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-1901716162
Emmm Arrow-rs also having one. You can regard it as a command line tool: https://github.com/apache/arrow-rs/blob/master/parquet/src/bin/parquet-concat.rs
Currently we didn't have a so like this🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-3921322763
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] efficiently combine parquet files [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-1897949393
cc @wgtmac
Does existing Java tool can do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-1899499283
> cc @wgtmac Does existing Java tool can do that?
Yes, please check https://github.com/apache/parquet-mr/tree/master/parquet-cli for `rewrite` command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39671:
URL: https://github.com/apache/arrow/issues/39671#issuecomment-3547086371
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][R] Efficiently combine parquet files [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #39671: [Parquet][R] Efficiently combine parquet files
URL: https://github.com/apache/arrow/issues/39671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org