You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/06/06 19:06:40 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #6569: [EPIC] Streaming partitioned writes
alamb opened a new issue, #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569
### Is your feature request related to a problem or challenge?
This is a tracking epic for a collection of features related to writing data.
The basic idea is better / full support for writing data:
1. to multiple (possibly Partitoned by value) files
2. To different file types (parquet, csv, json, avro, arrow)
3. In a streaming fashion (input doesn't need to be entirely buffered)
This is partially supported today programmatically (see [SessionContext::write_csv](https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.write_csv), etc)
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1683790637
An additional issue we should cut and add to this epic is allowing inserts to a sorted ListingTable. In the case of appending new files to a directory, I think it is as simple as having FileSinkExec require its input be sorted.
It can't really be supported efficiently for Append to existing file since it would require reading the existing file, sorting with the new data and rewriting the whole file. For this case, you could use insert overwrite instead if you really want to do this (which is another thing which we could cut a ticket to add support for).
Alternatively, we could have a check to see if 1) the table is sorted and 2) the input to FileSinkExec is sorted. If 1) is true but 2) is not, we would need to update the metadata about the table to indicate for subsequent queries it is no longer guaranteed to be sorted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1686809163
Filed https://github.com/apache/arrow-datafusion/issues/7354 to track
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1697405105
@alamb I opened #7442 to track adding support for parquet column level settings via SQL options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1680930199
> @alamb I opened https://github.com/apache/arrow-datafusion/issues/7298 to track improving statement level options/overrides relevant to this epic.
Thanks @devinjdangelo -- I added it to the list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Streaming partitioned writes [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1747927404
@alamb I made some progress on inserts to sorted tables https://github.com/apache/arrow-datafusion/issues/7354
This also got me thinking about inserts to partitioned tables, so I opened issue to track:
https://github.com/apache/arrow-datafusion/issues/7744
Lastly, I've been thinking we may want to deprecate and eventually remove the `SessionContext` write methods or alternatively hook them into `FileSinkExec` similarly to how the data frame write methods are set up.
https://github.com/apache/arrow-datafusion/blob/0408c2b1596417ba55a636fa3c8a601ffbdb0e60/datafusion/core/src/execution/context.rs#L1271-L1278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1677210854
FYI @devinjdangelo I updated this ticket with various issues related to the write code you are working on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1680457799
@alamb I opened #7298 to track improving statement level options/overrides relevant to this epic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] devinjdangelo commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1738316160
I added #7679 to track adding avro support for writes @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6569: [EPIC] Streaming partitioned writes
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1707045592
> @alamb I opened https://github.com/apache/arrow-datafusion/issues/7442 to track adding support for parquet column level settings via SQL options.
Thanks @devinjdangelo -- I added it to the list on this ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Streaming partitioned writes [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1749419336
> This also got me thinking about inserts to partitioned tables, so I opened issue to track: https://github.com/apache/arrow-datafusion/issues/7744
Thank you -- I added https://github.com/apache/arrow-datafusion/issues/7744 to the list on this ticket
> Lastly, I've been thinking we may want to deprecate and eventually remove the SessionContext write methods or alternatively hook them into FileSinkExec similarly to how the data frame write methods are set up.
That sounds like a reasonable idea to me. One challenge might be that the `SessionContext::write_csv` take an `ExecutionPlan` where the apis on DataFrame require a `LogicalPlan`.
Hooking them into `FileSinkExec` sounds like a great idea to consolidate the code. Shall I file a ticket for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Streaming partitioned writes [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #6569:
URL: https://github.com/apache/arrow-datafusion/issues/6569#issuecomment-1751696915
@alamb I wrote up an issue describing what we discussed on #7743 regarding empty files being written out and some potential solutions.
https://github.com/apache/arrow-datafusion/issues/7767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org