[GitHub] [iceberg] johnclara edited a comment on issue #1637: Spark Reads of Duplicate Datafiles

[GitBox <gi...@apache.org>]

johnclara edited a comment on issue #1637:
URL: https://github.com/apache/iceberg/issues/1637#issuecomment-718265869


   @rdblue our pipeline is:
   kafka topic -> akka streams service -> ingestion iceberg table -> spark compaction job -> compacted iceberg table
   
   the akka streams service only provides at least once delivery guarantees so we have to be okay with duplicate data anyways in the ingestion table
   
   the way we implemented our table's snapshot commit actually fails a lot due to DynamoDB rate limiting so we have to have retries around commits, which can lead to duplicate data files (not just duplicate rows)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org