[GitHub] [iceberg] aokolnychyi commented on pull request #1972: First version of the Iceberg sink for Apache Beam

[GitBox <gi...@apache.org>]

aokolnychyi commented on pull request #1972:
URL: https://github.com/apache/iceberg/pull/1972#issuecomment-794707731


   Thanks for the great work, @Fokko! 
   
   We have a person who is also interested in a Beam sink for Iceberg. It would be great to collaborate. @RussellSpitzer and I can help with reviews. I feel like this is not far from being ready.
   
   A couple of questions from me (I have just a basic understanding of Beam):
   - Is it correct that this PR supports only fixed windows in Beam? 
   - I head it is pretty common to write to multiple tables or determine which table to write to dynamically in Beam. Is it worth supporting such cases in the Iceberg sink? Does not have to be done in the first version, apparently.
   - Does Beam have any way to control the distribution and ordering of data that is passed for write? For example, we have added a way to ask Spark to distribute and order records according to the partition spec and sort order. A similar effort is in progress for Flink. The main reason is to reduce the number of produced files and order records accordingly to benefit from min/max skipping and better compression. Can people shuffle and sort data manually? Can we do that automatically?
   
   
   The first two points are minor but the last one can be a big deal at scale. For example, if we don't distribute and order data on write, we either need to produce a lot of small files or keep a lot of files open.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org