Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

["mas-chen (via GitHub)" <gi...@apache.org>]

mas-chen commented on code in PR #9173:
URL: https://github.com/apache/iceberg/pull/9173#discussion_r1411458771


##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java:
##########
@@ -197,17 +203,21 @@ private SplitEnumerator<IcebergSourceSplit, IcebergEnumeratorState> createEnumer
       LOG.info(
           "Iceberg source restored {} splits from state for table {}",
           enumState.pendingSplits().size(),
-          lazyTable().name());
+          tableName());
       assigner = assignerFactory.createAssigner(enumState.pendingSplits());
     }
 
+    // Create a copy of the table loader to avoid lifecycle management conflicts with the user
+    // provided table loader. This copy is only required for split planning, which uses the
+    // underlying io, and should be closed after split planning is complete

Review Comment:
   Just to keep it the same as `ContinuousSplitPlannerImpl` and `tableName()`, which perform the open/close themselves. and for the `ContinuousSplitPlannerImpl` we can't close it until the source is closed.
   
   Would it be more readable if the logic that requires a clone--abstract the clone, open, and then close? I think that's also fine but the original table loader needs to be closed by the source at the end of initialization



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org