You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2025/04/29 13:16:55 UTC
[I] Support parquet write from Arrow record batch [arrow-java]
luoyuxia opened a new issue, #735:
URL: https://github.com/apache/arrow-java/issues/735
### Describe the enhancement requested
I'm working on https://github.com/alibaba/fluss/issues/107 which enable convert Fluss arow structure data to Parquet directly but found the API missing in here.
Althogh #14151 supports to write from ArrowReader to file, it read from the ArrowReader, write and close the file direclty. But it's in a very coarse-grained , we almost have no control about the writing. Sometime, we want to control when to close the written parquet. Also it require `ArrowReader`, but if the arrow RecordBatch is read continuously from remote server . It's not easy to constuct a `ArrowReader`.
So, I think we may need to support the interface to write Arrow RecordBatch to Parquet via [virtual ::arrow::Status WriteRecordBatch(const ::arrow::RecordBatch& batch) = 0](https://github.com/apache/arrow/blob/c506b0806bd2b90410400d349a16bc4a5b1dd51c/cpp/src/parquet/arrow/writer.h#L125)
Just to as a show case, the api may look like:
```
public class ArrowBatchParquetWriter {
void write(RecordBatch recordbatch);
void close()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support parquet write from Arrow record batch [arrow-java]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #735:
URL: https://github.com/apache/arrow-java/issues/735#issuecomment-3489661943
> I think it's not out of the question, IIRC the concern when I've looked at it with colleagues before at various places is just that parquet-java is row-oriented and you can't get great performance out of it so it wasn't quite worth the effort, but if there's interested contributors I suppose that's not a problem
@lidavidm Hi, I open #904 to support it. I have verified it. Do you have time to review, or anyone can help review my pr? If so, I'll polish my pr to make it ready to review. The pr is almost ready, with only some code clean work remains to be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support parquet write from Arrow record batch [arrow-java]
Posted by "martin-traverse (via GitHub)" <gi...@apache.org>.
martin-traverse commented on issue #735:
URL: https://github.com/apache/arrow-java/issues/735#issuecomment-2927647261
Hello. I have been working on full support for Avro read / write, using the arrow-avro adapter. The intention is to provide a high-level API that allows reading / writing whole files block by block, with each block corresponding to one VSR and supporting VSR recycling.
I'd like to finish the Avro work first. But. Could we do something similar for Parquet? It would be a very simple read / write adapter for whole files, so querying / analytics etc would still want to use the Dataset code path. Of course there is a concern about maintenance overhead, but, I do think this is a gap and a solution would be useful. For example in my project we use Arrow-Java to translate to/from a range of formats, so naturally we want to include Parquet in that. Previously we've gone to the parquet-java API directly and had to build our own internal representation from there - a fair amount of work which would be much better handled in something like Arrow. Also there is the issue of Hadoop dependencies, I know the Parquet guys have been working to eliminate them but there is still quite a bit of fiddling needed avoid pulling those dependencies in - if we could package all that up into an Arrow adapter with a much smaller dependency tree, I think that would be valuabl
e as well.
I'd be happy to sketch out some ideas for review once the Avro work is complete. The parquet-java project already has code for schema translation which helps and somewhere I've got some old code from the last time I looked at this which might save some time as well.
Interested to know people's thoughts on this! In any case there are still one or two commits to go on Avro first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support parquet write from Arrow record batch [arrow-java]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #735:
URL: https://github.com/apache/arrow-java/issues/735#issuecomment-3455003831
> I think it's not out of the question, IIRC the concern when I've looked at it with colleagues before at various places is just that parquet-java is row-oriented and you can't get great performance out of it so it wasn't quite worth the effort, but if there's interested contributors I suppose that's not a problem
Hi, I'm interested in supporting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support parquet write from Arrow record batch [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #735:
URL: https://github.com/apache/arrow-java/issues/735#issuecomment-2928154585
I think it's not out of the question, IIRC the concern when I've looked at it with colleagues before at various places is just that parquet-java is row-oriented and you can't get great performance out of it so it wasn't quite worth the effort, but if there's interested contributors I suppose that's not a problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support parquet write from Arrow record batch [arrow-java]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia closed issue #735: Support parquet write from Arrow record batch
URL: https://github.com/apache/arrow-java/issues/735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org