You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/17 08:41:51 UTC

[GitHub] [iceberg] 0xffmeta opened a new pull request #4152: enable stream-results option to avoid driver oom

0xffmeta opened a new pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152


   When we try to run expire_snapshots procedure on the iceberg table for the first time, it often makes the driver OOM due to a big list of files to delete, which requires to increase the driver memory a lot. The iceberg table in our case contained more than ~2M data files and ~70K snapshots before we executed the expire_snapshots procedure. 
   
   We can allow the user to choose whether to use `stream-results` as a procedure option to avoid this case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809608339



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +84,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    Boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       Yeah - let me change it to primitive `boolean` and add the related test cases. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812489630



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java
##########
@@ -70,12 +70,13 @@
     extends BaseSparkAction<ExpireSnapshots, ExpireSnapshots.Result> implements ExpireSnapshots {
   private static final Logger LOG = LoggerFactory.getLogger(BaseExpireSnapshotsSparkAction.class);
 
+  // Make this variable public to be referenced for procedure option update

Review comment:
       Actually, what about moving this constant to `ExpireSnapshots`? What do you think, @RussellSpitzer? We do that in the action for rewriting data files, for instance. I am not sure how public we want to make this class.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta edited a comment on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta edited a comment on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1048486241


   @RussellSpitzer @aokolnychyi Thanks for the review and detailed comments. Please take another look for the PR - fixed the indention and variable names. 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809725261



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +84,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    Boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       @aokolnychyi Just updated the PR to include the suggested the change and test cases.
   For the test cases
   - the original `testUseLocalIterator` method might be a legacy test case (right now it uses `collectAsList` by default), so I renamed the method
   - Use `spark.sql.adaptive.enabled=false` conf to reduce the AQE impact since spark 3.2 enables this feature by default




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809721222



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       @kbendick just updated the PR to remove the conditional check. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1047909043


   I did another pass there are unfortunately a few things we need to clean up again, but ping me when it is ready and i will look again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812591560



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {

Review comment:
       I think we can hardcode the job number for localIterator, but I'm expecting this testcase to test out the different behavior for `stream-results` option.  If we hardcode the job number, we probably need to separate this into two testcases:
   
   1. use `toLocalIterator` with expected job number
   2. use `collectAsList` with expected job numer
   
   Does this make sense to you?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1055262456


   Thanks, @0xffmeta for the contribution and @RussellSpitzer @rdblue @kbendick for reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810404264



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       Oh, I think I remember that now that you mention it. It's really too bad. I believe there's a way to do that. I used it to avoid high peak memory consumption in broadcast joins a while ago. But that must not be part of the public API.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812592146



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +85,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       Agree - consistency is always good. 😃 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r813470499



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();

Review comment:
       Thanks for the suggestion 😄  - just updated the PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r813469387



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)

Review comment:
       Updated in the PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi edited a comment on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043386710


   This change looks good to me. Can we add a test that would invoke it, @0xffmeta?
   We have an example in `TestDeleteReachableFilesActions` on how such a flag can be tested.
   We should probably add something similar to `TestExpireSnapshotsProcedure`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] dramaticlly edited a comment on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
dramaticlly edited a comment on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043393780


   > @dramaticlly, it is doable today if you invoke the action directly and pass `stream-results` to it.
   
   Thank you @aokolnychyi , does it make sense to backport this as procedure for spark 3.1/3.0/2.4 as well? Not necessarily in this PR but question in general


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809725261



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +84,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    Boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       @aokolnychyi Just updated the PR to include the suggested the change and test cases.
   For the test cases
   - the original `testUseLocalIterator` method might be a legacy test case (right now it uses `collectAsList` by default), so I just renamed the method
   - Use `spark.sql.adaptive.enabled=false` conf to reduce the AQE impact since spark 3.2 enables this feature by default




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809683627



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       Would it be possible / desirable to just pass `streamResults` as is (either true or false), to avoid the conditional check? It should be an accepted option for the action either way.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1048356398


   Let me do another pass as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812489230



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java
##########
@@ -70,12 +70,13 @@
     extends BaseSparkAction<ExpireSnapshots, ExpireSnapshots.Result> implements ExpireSnapshots {
   private static final Logger LOG = LoggerFactory.getLogger(BaseExpireSnapshotsSparkAction.class);
 
+  // Make this variable public to be referenced for procedure option update

Review comment:
       +1




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1046435769


   Just rebase the commits to resolve a conflict in master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810392237



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1025,7 +1027,7 @@ public void testExpireAction() {
   }
 
   @Test
-  public void testUseLocalIterator() {
+  public void testUseCollectAsList() {

Review comment:
       This isn't needed, we can just fix the testUseLocalIterator test and not keep another version. This test was modified when we changed our Action api and it just wasn't updated correctly.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812052125



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();
 
-    Assert.assertEquals("Table does not have 1 snapshot after expiration", 1, Iterables.size(table.snapshots()));
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =
+              SparkActions.get().expireSnapshots(table).expireOlderThan(end)
+                      .option("stream-results", "false").execute();
 
-    int jobsAfter = spark.sparkContext().dagScheduler().nextJobId().get();
-    int totalJobsRun = jobsAfter - jobsBefore;
+      int jobsAfter = spark.sparkContext().dagScheduler().nextJobId().get();

Review comment:
       jobsAfterCollect




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi edited a comment on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043977157


   > does it make sense to backport this as procedure for spark 3.1/3.0/2.4 as well?
   
   @dramaticlly, if there is anyone in the community interested in backporting that after this is done, sure!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812492289



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +85,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       I know I suggested to convert it to `boolean` when I saw it being defaulted to `false`. However, I now remember why we use boxed primitives for other arguments: we want to inherit default values from the action. In this case, the procedure defines a default value. Instead, it should act as a wrapper that exposes certain configs from the underlying action. If those configs are not set, the action itself should pick a default value.
   
   It should probably become `Boolean` with null as the default value.
   
   ```
   Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
   
   ...
   
   if (streamResult != null) {
     action.option(ExpireSnapshots.STREAM_RESULTS, Boolean.toString(streamResult));
   }
   ```
   
   That way, the handling will be consistent with other properties and the procedure will inherit any changes in default action values that may happen in the future.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812490302



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();
 
-    Assert.assertEquals("Table does not have 1 snapshot after expiration", 1, Iterables.size(table.snapshots()));
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =

Review comment:
       The formatting is off in the entire file. Iceberg uses 2 spaces for identification and 4 spaces for continued identification (instead of 4 and 8 used in this file).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812980627



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {

Review comment:
       I don't know why we need one for collect, just to prove it is different when the option isn't enabled? I'm fine if you want to add that test as well, I just assumed that all the other tests are running in collect mode so you could just do a job count on any of them perhaps the simplest one?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043977157


   @dramaticlly, if there is anyone in the community interested in backporting that after this is done, sure!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810388634



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       toLocalIterator is much more expensive since every Spark task becomes a full Spark Job.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1045758240


   @RussellSpitzer Thanks for the review. I have updated the PR as suggested in the comment. 
   
   I'll backport this to other spark versions after this is merged. cc @dramaticlly @aokolnychyi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812487127



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -45,7 +46,8 @@
       ProcedureParameter.required("table", DataTypes.StringType),
       ProcedureParameter.optional("older_than", DataTypes.TimestampType),
       ProcedureParameter.optional("retain_last", DataTypes.IntegerType),
-      ProcedureParameter.optional("max_concurrent_deletes", DataTypes.IntegerType)
+      ProcedureParameter.optional("max_concurrent_deletes", DataTypes.IntegerType),
+      ProcedureParameter.optional("stream_results", DataTypes.BooleanType)

Review comment:
       I think the values are a bit different. We have to use `_` here while it is `-` in the option.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812044733



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -45,7 +46,8 @@
       ProcedureParameter.required("table", DataTypes.StringType),
       ProcedureParameter.optional("older_than", DataTypes.TimestampType),
       ProcedureParameter.optional("retain_last", DataTypes.IntegerType),
-      ProcedureParameter.optional("max_concurrent_deletes", DataTypes.IntegerType)
+      ProcedureParameter.optional("max_concurrent_deletes", DataTypes.IntegerType),
+      ProcedureParameter.optional("stream_results", DataTypes.BooleanType)

Review comment:
       The constant from the action can be used here as well




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812573080



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -45,7 +46,8 @@
       ProcedureParameter.required("table", DataTypes.StringType),
       ProcedureParameter.optional("older_than", DataTypes.TimestampType),
       ProcedureParameter.optional("retain_last", DataTypes.IntegerType),
-      ProcedureParameter.optional("max_concurrent_deletes", DataTypes.IntegerType)
+      ProcedureParameter.optional("max_concurrent_deletes", DataTypes.IntegerType),
+      ProcedureParameter.optional("stream_results", DataTypes.BooleanType)

Review comment:
       Yes - the parameter name is a little bit different between procedure and action.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1048486241


   @RussellSpitzer @aokolnychyi Thanks for the review and detailed comments. Please take another look for the PR - fixed the indention and variable names. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810390942



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       The reason for #1395 if anyone remembers is that having toLocalIterator on made this test suite take on the order of ten's of minutes.  I'm not sure what happened here though
   
   https://github.com/apache/iceberg/pull/1395/files#diff-c514ec43081677e77a31b19f98b424734da476b74043733d79f37c924b5dad5eR1031-R1032
   
   As the test that was renamed to "testCollectAsList" was actually using the local iterator before




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812489630



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java
##########
@@ -70,12 +70,13 @@
     extends BaseSparkAction<ExpireSnapshots, ExpireSnapshots.Result> implements ExpireSnapshots {
   private static final Logger LOG = LoggerFactory.getLogger(BaseExpireSnapshotsSparkAction.class);
 
+  // Make this variable public to be referenced for procedure option update

Review comment:
       Actually, what about moving this constant to `ExpireSnapshots`? What do you think, @RussellSpitzer? We do that in the action for rewriting data files, for instance.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812059644



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();
 
-    Assert.assertEquals("Table does not have 1 snapshot after expiration", 1, Iterables.size(table.snapshots()));
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =
+              SparkActions.get().expireSnapshots(table).expireOlderThan(end)
+                      .option("stream-results", "false").execute();
 
-    int jobsAfter = spark.sparkContext().dagScheduler().nextJobId().get();
-    int totalJobsRun = jobsAfter - jobsBefore;
+      int jobsAfter = spark.sparkContext().dagScheduler().nextJobId().get();
+      totalJobsRun.set(jobsAfter - jobsBefore);
 
-    checkExpirationResults(1L, 1L, 2L, results);
+      checkExpirationResults(1L, 1L, 2L, results);
+    });
+
+    table.newOverwrite()
+            .deleteFile(FILE_C)
+            .addFile(FILE_B)
+            .commit();
+
+    long endAgain = rightAfterSnapshot();
+
+    int jobsBeforeAgain = spark.sparkContext().dagScheduler().nextJobId().get();

Review comment:
       JobsBeforeStreamResults, etc..

##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();
 
-    Assert.assertEquals("Table does not have 1 snapshot after expiration", 1, Iterables.size(table.snapshots()));
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =
+              SparkActions.get().expireSnapshots(table).expireOlderThan(end)
+                      .option("stream-results", "false").execute();
 
-    int jobsAfter = spark.sparkContext().dagScheduler().nextJobId().get();
-    int totalJobsRun = jobsAfter - jobsBefore;
+      int jobsAfter = spark.sparkContext().dagScheduler().nextJobId().get();
+      totalJobsRun.set(jobsAfter - jobsBefore);
 
-    checkExpirationResults(1L, 1L, 2L, results);
+      checkExpirationResults(1L, 1L, 2L, results);
+    });
+
+    table.newOverwrite()
+            .deleteFile(FILE_C)
+            .addFile(FILE_B)
+            .commit();
+
+    long endAgain = rightAfterSnapshot();
+
+    int jobsBeforeAgain = spark.sparkContext().dagScheduler().nextJobId().get();
+
+    AtomicReference<Integer> totalJobsRunAgain = new AtomicReference<>();
+
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =
+              SparkActions.get().expireSnapshots(table).expireOlderThan(endAgain)

Review comment:
       Formatting is incorrect here as well




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812051836



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();

Review comment:
       Maybe rename this "jobsBeforeCollect"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] dramaticlly commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
dramaticlly commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043385584


   Just for my own curiosity, is it doable today by setting the table properties as in https://github.com/apache/iceberg/blob/master/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java#L215-L217


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043388374


   @dramaticlly, it is doable today if you invoke the action directly and pass `stream-results` to it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1050536239


   @RussellSpitzer @aokolnychyi I have updated the PR based on the comment. Please take another look when you have some time. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1055296497


   Many thanks for the review, @aokolnychyi @RussellSpitzer @rdblue @kbendick!
   Let me submit another PR to backport this ot other spark versions. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809702325



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       @rdblue, in most cases, calling `toLocalIterator` would be much more expensive as it would trigger multiple jobs (as far as I remember). We started with that but then changed in the PR that @0xffmeta linked. Tables should be maintained regularly so that's not an issue in most cases.
   
   @RussellSpitzer may have more context.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809725261



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +84,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    Boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       @aokolnychyi Just updated the PR to include the suggested the change and test cases.
   For the original test case `testUseLocalIterator`:
   - the original `testUseLocalIterator` method might be a legacy test case (right now it uses `collectAsList` by default), so I renamed the method to `testUseCollectAsList`
   - Use `spark.sql.adaptive.enabled=false` conf to reduce the AQE impact since spark 3.2 enables this feature by default




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810392559



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,8 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      action.option("stream-results", Boolean.toString(streamResult));

Review comment:
       We should probably make "stream-results" a constant in the ExpireSnapshotsAction and use it here rather than passing it as a string




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1047657437


   @RussellSpitzer Could you help to merge this PR so that I don't need to rebase the commit for potential conflict? 🥲 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812052608



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();

Review comment:
       jobsRunDuringCollect




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r813467630



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {

Review comment:
       Understood - indeed, `collect` is already tested multiple times in other tests, and it's not necessary to repeat it again. :) 
   And I have updated the test case to only check against the expected the job numbers. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812490302



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();
 
-    Assert.assertEquals("Table does not have 1 snapshot after expiration", 1, Iterables.size(table.snapshots()));
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =

Review comment:
       The formatting is off in this entire file. Iceberg uses 2 spaces for identification and 4 spaces for continued identification (instead of 4 and 8 used in this file).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812591858



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)
+            .commit();
 
     table.newOverwrite()
-        .deleteFile(FILE_A)
-        .addFile(FILE_B)
-        .commit();
+            .deleteFile(FILE_A)
+            .addFile(FILE_B)
+            .commit();
 
     table.newFastAppend()
-        .appendFile(FILE_C)
-        .commit();
+            .appendFile(FILE_C)
+            .commit();
 
     long end = rightAfterSnapshot();
 
     int jobsBefore = spark.sparkContext().dagScheduler().nextJobId().get();
 
-    ExpireSnapshots.Result results =
-        SparkActions.get().expireSnapshots(table).expireOlderThan(end).execute();
+    AtomicReference<Integer> totalJobsRun = new AtomicReference<>();
 
-    Assert.assertEquals("Table does not have 1 snapshot after expiration", 1, Iterables.size(table.snapshots()));
+    withSQLConf(ImmutableMap.of("spark.sql.adaptive.enabled", "false"), () -> {
+      ExpireSnapshots.Result results =

Review comment:
       Thanks for the explanation. Just updated the indention. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi merged pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi merged pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi edited a comment on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043386710


   This change looks good to me (one nit). Can we add a test that would invoke it, @0xffmeta?
   We have an example in `TestDeleteReachableFilesActions` on how such a flag can be tested.
   We should probably add something similar to `TestExpireSnapshotsProcedure`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] dramaticlly commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
dramaticlly commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043393780


   > @dramaticlly, it is doable today if you invoke the action directly and pass `stream-results` to it.
   
   Thank you Anton, does it make sense to backport this as procedure for spark 3.1/3.0/2.4 as well? Not necessarily in this PR but question in general


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#issuecomment-1043386710


   This change looks good to me. Can we add a test that would invoke it, @0xffmeta?
   We have an example in `TestDeleteReachableFilesActions` on how such a flag can be tested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809606433



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       There is one [PR](https://github.com/apache/iceberg/pull/1395) previously to collect all the results by default. This should be most usual cases. 
   The `stream-results` option should be explicitly enabled for the case that contains huge amount of data files. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809434283



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -83,6 +84,7 @@ public StructType outputType() {
     Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
     Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
     Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
+    Boolean streamResult = args.isNullAt(4) ? false : args.getBoolean(4);

Review comment:
       nit: can this be a primitive `boolean`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809499641



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       @aokolnychyi, why not use this by default?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r809702325



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       @rdblue, in most cases, calling `toLocalIterator` would be much more expensive as it would trigger multiple jobs (as far as I remember). We started with that but then changed in the PR that @0xffmeta linked.
   
   @RussellSpitzer may have more context.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812492614



##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestExpireSnapshotsProcedure.java
##########
@@ -283,4 +283,31 @@ public void testExpireDeleteFiles() throws Exception {
     Assert.assertFalse("Delete manifest should be removed", localFs.exists(deleteManifestPath));
     Assert.assertFalse("Delete file should be removed", localFs.exists(deleteFilePath));
   }
+
+  @Test
+  public void testExpireSnapshotWithStreamResultsEnabled() {
+    sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
+
+    sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
+    sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
+
+    waitUntilAfter(table.currentSnapshot().timestampMillis());
+
+    Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
+
+    List<Object[]> output = sql(

Review comment:
       The formatting is off here too. Should be 4 spaces instead of 8 for continued indentation.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] 0xffmeta commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
0xffmeta commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812580046



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java
##########
@@ -70,12 +70,13 @@
     extends BaseSparkAction<ExpireSnapshots, ExpireSnapshots.Result> implements ExpireSnapshots {
   private static final Logger LOG = LoggerFactory.getLogger(BaseExpireSnapshotsSparkAction.class);
 
+  // Make this variable public to be referenced for procedure option update

Review comment:
       I think putting the constant into `ExpireSnapshots` might be a good idea, and also I noticed that actions are all base class, e.g. `BaseExpireSnapshotsSparkAction`. Does this mean that the behavior can be overwritten in future? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810393248



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,8 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      action.option("stream-results", Boolean.toString(streamResult));

Review comment:
       It already is :) It's just private, while I wish we could just make it package private on iceberg.spark I think it's fine to just make it public in BaseExpireSnapshotsSparkAction




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810392237



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1025,7 +1027,7 @@ public void testExpireAction() {
   }
 
   @Test
-  public void testUseLocalIterator() {
+  public void testUseCollectAsList() {

Review comment:
       This isn't needed, we can just fix the testUseLocalIterator test and not keep another version. This test was modified when we changed our Action api and it just wasn't updated correctly. So drop "testUseCollectAsList" and keep "testUseLocalIterator"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r810391822



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java
##########
@@ -102,6 +104,10 @@ public StructType outputType() {
         action.executeDeleteWith(executorService(maxConcurrentDeletes, "expire-snapshots"));
       }
 
+      if (streamResult) {
+        action.option("stream-results", streamResult.toString());

Review comment:
       Oh I see we lost it when we switched the APIs #3469 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812045214



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseExpireSnapshotsSparkAction.java
##########
@@ -70,12 +70,13 @@
     extends BaseSparkAction<ExpireSnapshots, ExpireSnapshots.Result> implements ExpireSnapshots {
   private static final Logger LOG = LoggerFactory.getLogger(BaseExpireSnapshotsSparkAction.class);
 
+  // Make this variable public to be referenced for procedure option update

Review comment:
       I don't think this comment is really necessary, I think it's fine to just let it be a public static final string 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812044486



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {
     table.newFastAppend()
-        .appendFile(FILE_A)
-        .commit();
+            .appendFile(FILE_A)

Review comment:
       nit: Formatting changed here and in the below examples




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4152: enable stream-results option to avoid spark driver oom

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4152:
URL: https://github.com/apache/iceberg/pull/4152#discussion_r812062844



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestExpireSnapshotsAction.java
##########
@@ -1027,34 +1029,59 @@ public void testExpireAction() {
   @Test
   public void testUseLocalIterator() {

Review comment:
       In this test would it be possible to just hardcode the expected number of localiterator jobs? Or is this variable? If we don't expect the number of jobs for the "collect"version to change I feel like this can be safely hardcoded.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org