You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/04/08 18:50:26 UTC
[I] Add support for native Parquet writes [datafusion-comet]
andygrove opened a new issue, #1625:
URL: https://github.com/apache/datafusion-comet/issues/1625
### What is the problem the feature request solves?
With ETL queries that read from Parquet, perform a transformation, and then write back out to Parquet, we currently have to perform a columnar-to-row conversion before the write.
If we add support for DataFusion's Parquet writer then we can avoid the columnar-to-row conversion.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for native Parquet writes [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1625:
URL: https://github.com/apache/datafusion-comet/issues/1625#issuecomment-3563798491
I updated the issue with Claude's suggestions for implementing this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Add support for native Parquet writes [datafusion-comet]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on issue #1625:
URL: https://github.com/apache/datafusion-comet/issues/1625#issuecomment-4435581291
Hey Andy! I just did a bit of research here, but from an iceberg-specific focus.
I have a plan drafted out, which is essentially pretty restrictive about when we'll actually do the writes (parquet only, iceberg-rust needs all the same features, etc etc) and I hope to try and get up a PR fairly soon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for native Parquet writes [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1625:
URL: https://github.com/apache/datafusion-comet/issues/1625#issuecomment-3030362983
> Hello [@andygrove](https://github.com/andygrove) , this sounds like an interesting problem I'd be keen to work on. I think I might need some guidance to navigate it effectively. Would you be open to chatting on Slack or Discord while I'm working on it?
@dharanad I'm happy to chat, but I'm not sure that I know enough about implementing this without some research myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org