[GitHub] [incubator-iceberg] prabcs commented on issue #179: Use Iceberg tables as sources for Spark Structured Streaming

[GitBox <gi...@apache.org>]

prabcs commented on issue #179: Use Iceberg tables as sources for Spark Structured Streaming
URL: https://github.com/apache/incubator-iceberg/issues/179#issuecomment-513884312
 
 
   👋, I'm going through iceberg and it seems quite interesting. I have couple of questions on this issue.
   
   > What do we consider as one batch?
   
   With Spark's Structured Streaming, need to consider the size of a microbatch as well. The size of a microbatch can be defined at the query level. Typically, that would mean reading that number of records from the stream (FileStream) in this case. The [File Source](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala) in Spark Structured Streaming can read new files per batch. So, the design of a streaming Iceberg File Source can be based on that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org