[GitHub] [iceberg] SreeramGarlapati commented on a change in pull request #2611: [WorkInProgress] Spark3 structured streaming micro_batch read support

[GitBox <gi...@apache.org>]

SreeramGarlapati commented on a change in pull request #2611:
URL: https://github.com/apache/iceberg/pull/2611#discussion_r635010805



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java
##########
@@ -159,8 +159,14 @@ private Schema schemaWithMetadataColumns() {
 
   @Override
   public Scan build() {
-    return new SparkBatchQueryScan(
-        spark, table, caseSensitive, schemaWithMetadataColumns(), filterExpressions, options);
+    // TODO: understand how to differentiate that this is a spark streaming microbatch scan.
+    if (false) {

Review comment:
       @aokolnychyi / @RussellSpitzer 
   Spark3 gives ScanBuilder - abstraction - to define all types of Scans (Batch, MicroBatch & Continuous). But, the current implementation / class modelling - has SparkBatchScan as the Scan implementation.
   Looking at some of the concerns of BatchScan - all the way from the State maintenance of a single SnapshotId to read from, the asOfTimeStamp & features like VectorizedReads - all of these don't seem relevant to Streaming Scans.
   So, I feel that we need to divide out Streaming Scans into a different class.
   Does this thought process - make sense?
   If we go by this route - do you folks know - how to pass different Scan objects to Spark based on Batch vs Streaming?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org