You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hemanthboyina (via GitHub)" <gi...@apache.org> on 2026/09/07 18:29:06 UTC

[PR] Spark: Synchronize runtime filtering to fix row duplication in COW UPDATE [iceberg]

hemanthboyina opened a new pull request, #18005:
URL: https://github.com/apache/iceberg/pull/18005

   A copy-on-write `UPDATE ... WHERE ... IN/EXISTS (<subquery>)` is rewritten by Spark into a UNION whose two branches share the same Iceberg scan. Under AQE, Spark runs the `filter(Predicate[])` runtime-filter callback on that shared scan from multiple threads at once.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4200269014


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.Timeout;
+
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final long TIMEOUT_SECONDS = 10;
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  @TestTemplate
+  @Timeout(value = 60, unit = TimeUnit.SECONDS)
+  public void concurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql("CREATE TABLE %s (id BIGINT, data STRING) USING iceberg", tableName);
+    sql("INSERT INTO %s VALUES (1, 'a')", tableName);
+    sql("INSERT INTO %s VALUES (2, 'b')", tableName);
+
+    Table table = validationCatalog.loadTable(tableIdent);
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseFilter = new CountDownLatch(1);
+    InstrumentedScan scan = newScan(table, enteredResetTasks, releaseFilter);
+
+    assertThat(scan.tasks()).hasSize(2);
+    scan.taskGroups();
+    String location = scan.tasks().get(0).file().location();
+    NamedReference fileRef = FieldReference.apply(MetadataColumns.FILE_PATH.name());
+    LiteralValue<UTF8String> literal =
+        new LiteralValue<>(UTF8String.fromString(location), DataTypes.StringType);
+    Predicate[] predicates = {
+      new Predicate(
+          "IN", new org.apache.spark.sql.connector.expressions.Expression[] {fileRef, literal})
+    };
+
+    FutureTask<Void> winnerTask =
+        new FutureTask<>(
+            () -> {
+              scan.filter(predicates);
+              return null;
+            });
+    FutureTask<int[]> loserTask =
+        new FutureTask<>(
+            () -> {
+              scan.filter(predicates);
+              int tasks = scan.tasks().size();
+              int taskGroups =
+                  scan.taskGroups().stream().mapToInt(group -> group.tasks().size()).sum();
+              return new int[] {tasks, taskGroups};

Review Comment:
   Could we assert the file paths here, checking that both `tasks()` and the flattened `taskGroups()` contain exactly `location`? We could also assert that the initial task groups contain both original files. This would make the test a little more generic and useful overall by checking which files survive the filter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4208021773


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.Timeout;
+
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final long TIMEOUT_SECONDS = 10;
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  @TestTemplate
+  @Timeout(value = 60, unit = TimeUnit.SECONDS)
+  public void concurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql("CREATE TABLE %s (id BIGINT, data STRING) USING iceberg", tableName);
+    sql("INSERT INTO %s VALUES (1, 'a')", tableName);
+    sql("INSERT INTO %s VALUES (2, 'b')", tableName);
+
+    Table table = validationCatalog.loadTable(tableIdent);
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseFilter = new CountDownLatch(1);
+    InstrumentedScan scan = newScan(table, enteredResetTasks, releaseFilter);
+
+    assertThat(scan.tasks()).hasSize(2);
+    scan.taskGroups();
+    String location = scan.tasks().get(0).file().location();
+    NamedReference fileRef = FieldReference.apply(MetadataColumns.FILE_PATH.name());
+    LiteralValue<UTF8String> literal =
+        new LiteralValue<>(UTF8String.fromString(location), DataTypes.StringType);
+    Predicate[] predicates = {
+      new Predicate(
+          "IN", new org.apache.spark.sql.connector.expressions.Expression[] {fileRef, literal})
+    };
+
+    FutureTask<Void> winnerTask =
+        new FutureTask<>(
+            () -> {
+              scan.filter(predicates);
+              return null;
+            });
+    FutureTask<int[]> loserTask =
+        new FutureTask<>(
+            () -> {
+              scan.filter(predicates);
+              int tasks = scan.tasks().size();
+              int taskGroups =
+                  scan.taskGroups().stream().mapToInt(group -> group.tasks().size()).sum();
+              return new int[] {tasks, taskGroups};

Review Comment:
   done, added the changes, also rebased to fix the compilation issue



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4198348338


##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.Timeout;
+
+/** Tests concurrent runtime filtering on a shared copy-on-write scan. */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 2;
+  private static final long TIMEOUT_SECONDS = 10;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery unions two branches over one shared scan, filtered concurrently
+  // under AQE; the losing branch must see the narrowed task set, not the full one.
+  @TestTemplate
+  @Timeout(value = 60, unit = TimeUnit.SECONDS)
+  public void concurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner did not enter resetTasks")
+        .isTrue();
+
+    loser.start();
+    assertThat(awaitLoserSettled(loser, observedTaskCount))
+        .as("loser did not block on the scan monitor or record a result")
+        .isTrue();
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+    assertThat(winner.isAlive()).as("winner did not terminate").isFalse();
+    assertThat(loser.isAlive()).as("loser did not terminate").isFalse();
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as("losing branch must observe the narrowed single-file task set, not the full set")
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  private static boolean awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return true;
+      }
+      Thread.sleep(1);
+    }
+    return false;
+  }
+
+  // mirrors SparkScanBuilder#buildCopyOnWriteScan but returns an instrumented subclass

Review Comment:
   agreed, removed the comments and modified the tests 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho merged PR #18005:
URL: https://github.com/apache/iceberg/pull/18005


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r3995540786


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/**
+ * Reproduces the copy-on-write runtime-filter concurrency bug (issue #18004).
+ *
+ * <p>A COW {@code UPDATE ... WHERE EXISTS (<subquery>)} is rewritten as {@code Union(Filter(cond,
+ * S), Filter(NOT cond, S))} where BOTH branches share the same {@link SparkCopyOnWriteScan} {@code
+ * S}. Under AQE the two branch stages are prepared concurrently, so the {@link
+ * org.apache.spark.sql.connector.read.SupportsRuntimeV2Filtering#filter(Predicate[])} callback runs
+ * concurrently on the one shared scan. {@code filter()} is a non-atomic check-then-act: it
+ * publishes {@code filteredLocations} first, then narrows {@code tasks()}, and only at the very end
+ * calls {@code resetTasks(...)}. A losing branch that observes {@code filteredLocations} already
+ * set (so it skips narrowing) while the winner has not yet reached {@code resetTasks} would read
+ * the planning-time memoized FULL task set and scan every file, duplicating rows on commit.
+ *
+ * <p>This drives the race deterministically: a winner thread narrows the scan but parks inside an
+ * un-synchronized {@code resetTasks} override, while a loser thread runs {@code filter()} and reads
+ * the shared task set. With {@code filter()} synchronized the loser is serialized behind the winner
+ * and observes the narrowed single-file set; revert the {@code synchronized} keyword and it
+ * observes the full set.
+ */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long PARK_MILLIS = TimeUnit.SECONDS.toMillis(2);
+  private static final long LATCH_TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    // one single-row data file per INSERT, so the full task set is clearly distinguishable from
+    // the single-file set the runtime filter should produce
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks);
+
+    // memoize the full planning-time task set and task groups exactly as query planning would
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    // `_file IN (<one location>)` runtime predicate, matching what Spark passes for a COW filter
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    // winner: narrows the scan but parks inside resetTasks. It holds the scan monitor while it
+    // sleeps ONLY if filter() is synchronized; that is exactly the production behavior under test.
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+                enteredResetTasks.countDown();
+              }
+            },
+            "cow-filter-winner");
+
+    // loser: proceeds only once the winner has published filteredLocations and entered resetTasks.
+    // Its own filter() short-circuits (the guard fails for the same single location), then it reads
+    // the shared task set - which must already be narrowed.
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                assertThat(enteredResetTasks.await(LATCH_TIMEOUT_SECONDS, TimeUnit.SECONDS))
+                    .as("winner should have entered resetTasks")
+                    .isTrue();
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    loser.start();
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as(
+            "the losing UNION branch must observe the narrowed single-file task set, never the "
+                + "full %s-file set (which would rewrite every file and duplicate rows, issue "
+                + "#18004)",
+            FILE_COUNT)
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  // mirrors SparkScanBuilder#buildCopyOnWriteScan (non-null snapshot path) but returns an
+  // instrumented subclass so the race window inside resetTasks can be controlled deterministically
+  private InstrumentedCopyOnWriteScan newInstrumentedScan(
+      Table table, CountDownLatch enteredResetTasks) {
+    SparkReadConf readConf = new SparkReadConf(spark, table);
+    Schema schema = table.schema();
+    Snapshot snapshot = table.currentSnapshot();
+
+    BatchScan batchScan =
+        table
+            .newBatchScan()
+            .useSnapshot(snapshot.snapshotId())
+            .ignoreResiduals()
+            .caseSensitive(readConf.caseSensitive())
+            .filter(Expressions.alwaysTrue())
+            .project(schema);
+
+    return new InstrumentedCopyOnWriteScan(
+        spark,
+        table,
+        schema,
+        snapshot,
+        null /* branch */,
+        batchScan,
+        readConf,
+        schema /* projection */,
+        Lists.newArrayList(),
+        () -> null,
+        enteredResetTasks);
+  }
+
+  private static Predicate filePathInPredicate(String location) {
+    NamedReference fileRef = FieldReference.apply(MetadataColumns.FILE_PATH.name());
+    LiteralValue<UTF8String> literal =
+        new LiteralValue<>(UTF8String.fromString(location), DataTypes.StringType);
+    return new Predicate(
+        "IN", new org.apache.spark.sql.connector.expressions.Expression[] {fileRef, literal});
+  }
+
+  private static int countFiles(List<ScanTaskGroup<FileScanTask>> taskGroups) {
+    int count = 0;
+    for (ScanTaskGroup<FileScanTask> group : taskGroups) {
+      count += group.tasks().size();
+    }
+    return count;
+  }
+
+  /**
+   * A {@link SparkCopyOnWriteScan} that parks inside {@code resetTasks} (before delegating to the
+   * real implementation) so the check-then-act window in {@code filter()} can be observed by
+   * another thread. The override is intentionally NOT synchronized: the only serialization under
+   * test must come from {@code SparkCopyOnWriteScan#filter}, so reverting that fix reliably
+   * reproduces the race instead of being masked here.
+   */
+  private static class InstrumentedCopyOnWriteScan extends SparkCopyOnWriteScan {
+    private final CountDownLatch enteredResetTasks;
+
+    InstrumentedCopyOnWriteScan(
+        SparkSession spark,
+        Table table,
+        Schema schema,
+        Snapshot snapshot,
+        String branch,
+        BatchScan scan,
+        SparkReadConf readConf,
+        Schema projection,
+        List<Expression> filters,
+        Supplier<ScanReport> scanReportSupplier,
+        CountDownLatch enteredResetTasks) {
+      super(
+          spark,
+          table,
+          schema,
+          snapshot,
+          branch,
+          scan,
+          readConf,
+          projection,
+          filters,
+          scanReportSupplier);
+      this.enteredResetTasks = enteredResetTasks;
+    }
+
+    @Override
+    protected void resetTasks(List<FileScanTask> filteredTasks) {
+      enteredResetTasks.countDown();
+      try {
+        Thread.sleep(PARK_MILLIS);

Review Comment:
   done, dropped the fixed sleep



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4061769541


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */

Review Comment:
   done



##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r3973277785


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java:
##########
@@ -124,7 +124,10 @@ public NamedReference[] filterAttributes() {
   }
 
   @Override
-  public void filter(Predicate[] predicates) {
+  // synchronized on the same monitor as tasks()/taskGroups()/resetTasks() so the non-atomic

Review Comment:
   can we simplify these AI comments?  ie, check-then-act?  Maybe just explain what field is protected?



##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/**
+ * Reproduces the copy-on-write runtime-filter concurrency bug (issue #18004).
+ *
+ * <p>A COW {@code UPDATE ... WHERE EXISTS (<subquery>)} is rewritten as {@code Union(Filter(cond,

Review Comment:
   In the interest of organization can we move this whole javadoc to the method, and maybe just link the issue and a few basic details instead of re-gurgitating the whole issue description again?  It's quite spark detail heavy



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -118,7 +118,12 @@ public NamedReference[] filterAttributes() {
   }
 
   @Override
-  public void filter(Predicate[] predicates) {
+  // synchronized on the same monitor as tasks()/taskGroups()/resetTasks() so the non-atomic

Review Comment:
   same for these and following comments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r3995541882


##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/**
+ * Reproduces the copy-on-write runtime-filter concurrency bug (issue #18004).
+ *
+ * <p>A COW {@code UPDATE ... WHERE EXISTS (<subquery>)} is rewritten as {@code Union(Filter(cond,

Review Comment:
   resolved it too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4061772564


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery becomes a UNION whose branches share one scan, which Spark can
+  // filter() concurrently under AQE. Without synchronization a branch reads the full task set and
+  // rewrites every file, duplicating rows (#18004).
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {

Review Comment:
   done, renamed to concurrentRuntimeFilteringNarrowsSharedScan



##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery becomes a UNION whose branches share one scan, which Spark can
+  // filter() concurrently under AQE. Without synchronization a branch reads the full task set and
+  // rewrites every file, duplicating rows (#18004).
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    // one single-row file per INSERT, so the narrowed set (1) is clearly distinct from the full set
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    // memoize the full task set and groups exactly as query planning would
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    // `_file IN (<one location>)` runtime predicate, matching what Spark passes for a COW filter
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    // winner: narrows the scan, then parks in resetTasks while holding the scan monitor
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    // loser: runs filter() (short-circuits) then reads the shared task set, which must be narrowed
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner should have entered resetTasks")
+        .isTrue();
+
+    loser.start();
+    // release the winner once the loser has blocked on the monitor or finished its stale read
+    awaitLoserSettled(loser, observedTaskCount);
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as(
+            "the losing UNION branch must observe the narrowed single-file task set, never the "
+                + "full %s-file set (which would rewrite every file and duplicate rows, issue "
+                + "#18004)",
+            FILE_COUNT)
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  // waits until the loser can progress no further before releasing the winner: either blocked on
+  // the scan monitor (synchronized filter()) or already done recording its task count
+  private static void awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return;
+      }
+      Thread.sleep(1);
+    }
+  }

Review Comment:
   done too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r3972730691


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/**
+ * Reproduces the copy-on-write runtime-filter concurrency bug (issue #18004).
+ *
+ * <p>A COW {@code UPDATE ... WHERE EXISTS (<subquery>)} is rewritten as {@code Union(Filter(cond,
+ * S), Filter(NOT cond, S))} where BOTH branches share the same {@link SparkCopyOnWriteScan} {@code
+ * S}. Under AQE the two branch stages are prepared concurrently, so the {@link
+ * org.apache.spark.sql.connector.read.SupportsRuntimeV2Filtering#filter(Predicate[])} callback runs
+ * concurrently on the one shared scan. {@code filter()} is a non-atomic check-then-act: it
+ * publishes {@code filteredLocations} first, then narrows {@code tasks()}, and only at the very end
+ * calls {@code resetTasks(...)}. A losing branch that observes {@code filteredLocations} already
+ * set (so it skips narrowing) while the winner has not yet reached {@code resetTasks} would read
+ * the planning-time memoized FULL task set and scan every file, duplicating rows on commit.
+ *
+ * <p>This drives the race deterministically: a winner thread narrows the scan but parks inside an
+ * un-synchronized {@code resetTasks} override, while a loser thread runs {@code filter()} and reads
+ * the shared task set. With {@code filter()} synchronized the loser is serialized behind the winner
+ * and observes the narrowed single-file set; revert the {@code synchronized} keyword and it
+ * observes the full set.
+ */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long PARK_MILLIS = TimeUnit.SECONDS.toMillis(2);
+  private static final long LATCH_TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    // one single-row data file per INSERT, so the full task set is clearly distinguishable from
+    // the single-file set the runtime filter should produce
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks);
+
+    // memoize the full planning-time task set and task groups exactly as query planning would
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    // `_file IN (<one location>)` runtime predicate, matching what Spark passes for a COW filter
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    // winner: narrows the scan but parks inside resetTasks. It holds the scan monitor while it
+    // sleeps ONLY if filter() is synchronized; that is exactly the production behavior under test.
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+                enteredResetTasks.countDown();
+              }
+            },
+            "cow-filter-winner");
+
+    // loser: proceeds only once the winner has published filteredLocations and entered resetTasks.
+    // Its own filter() short-circuits (the guard fails for the same single location), then it reads
+    // the shared task set - which must already be narrowed.
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                assertThat(enteredResetTasks.await(LATCH_TIMEOUT_SECONDS, TimeUnit.SECONDS))
+                    .as("winner should have entered resetTasks")
+                    .isTrue();
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    loser.start();
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as(
+            "the losing UNION branch must observe the narrowed single-file task set, never the "
+                + "full %s-file set (which would rewrite every file and duplicate rows, issue "
+                + "#18004)",
+            FILE_COUNT)
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  // mirrors SparkScanBuilder#buildCopyOnWriteScan (non-null snapshot path) but returns an
+  // instrumented subclass so the race window inside resetTasks can be controlled deterministically
+  private InstrumentedCopyOnWriteScan newInstrumentedScan(
+      Table table, CountDownLatch enteredResetTasks) {
+    SparkReadConf readConf = new SparkReadConf(spark, table);
+    Schema schema = table.schema();
+    Snapshot snapshot = table.currentSnapshot();
+
+    BatchScan batchScan =
+        table
+            .newBatchScan()
+            .useSnapshot(snapshot.snapshotId())
+            .ignoreResiduals()
+            .caseSensitive(readConf.caseSensitive())
+            .filter(Expressions.alwaysTrue())
+            .project(schema);
+
+    return new InstrumentedCopyOnWriteScan(
+        spark,
+        table,
+        schema,
+        snapshot,
+        null /* branch */,
+        batchScan,
+        readConf,
+        schema /* projection */,
+        Lists.newArrayList(),
+        () -> null,
+        enteredResetTasks);
+  }
+
+  private static Predicate filePathInPredicate(String location) {
+    NamedReference fileRef = FieldReference.apply(MetadataColumns.FILE_PATH.name());
+    LiteralValue<UTF8String> literal =
+        new LiteralValue<>(UTF8String.fromString(location), DataTypes.StringType);
+    return new Predicate(
+        "IN", new org.apache.spark.sql.connector.expressions.Expression[] {fileRef, literal});
+  }
+
+  private static int countFiles(List<ScanTaskGroup<FileScanTask>> taskGroups) {
+    int count = 0;
+    for (ScanTaskGroup<FileScanTask> group : taskGroups) {
+      count += group.tasks().size();
+    }
+    return count;
+  }
+
+  /**
+   * A {@link SparkCopyOnWriteScan} that parks inside {@code resetTasks} (before delegating to the
+   * real implementation) so the check-then-act window in {@code filter()} can be observed by
+   * another thread. The override is intentionally NOT synchronized: the only serialization under
+   * test must come from {@code SparkCopyOnWriteScan#filter}, so reverting that fix reliably
+   * reproduces the race instead of being masked here.
+   */
+  private static class InstrumentedCopyOnWriteScan extends SparkCopyOnWriteScan {
+    private final CountDownLatch enteredResetTasks;
+
+    InstrumentedCopyOnWriteScan(
+        SparkSession spark,
+        Table table,
+        Schema schema,
+        Snapshot snapshot,
+        String branch,
+        BatchScan scan,
+        SparkReadConf readConf,
+        Schema projection,
+        List<Expression> filters,
+        Supplier<ScanReport> scanReportSupplier,
+        CountDownLatch enteredResetTasks) {
+      super(
+          spark,
+          table,
+          schema,
+          snapshot,
+          branch,
+          scan,
+          readConf,
+          projection,
+          filters,
+          scanReportSupplier);
+      this.enteredResetTasks = enteredResetTasks;
+    }
+
+    @Override
+    protected void resetTasks(List<FileScanTask> filteredTasks) {
+      enteredResetTasks.countDown();
+      try {
+        Thread.sleep(PARK_MILLIS);

Review Comment:
   Could we replace this fixed sleep with explicit coordination? For example, keep the winner parked on a release latch, start the loser, wait until it is either blocked on the scan monitor or has completed its stale read, then release the winner. If the loser is not scheduled within these two seconds, this test could pass even after removing `synchronized`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4019776924


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery becomes a UNION whose branches share one scan, which Spark can
+  // filter() concurrently under AQE. Without synchronization a branch reads the full task set and
+  // rewrites every file, duplicating rows (#18004).
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    // one single-row file per INSERT, so the narrowed set (1) is clearly distinct from the full set
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    // memoize the full task set and groups exactly as query planning would
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    // `_file IN (<one location>)` runtime predicate, matching what Spark passes for a COW filter
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    // winner: narrows the scan, then parks in resetTasks while holding the scan monitor
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    // loser: runs filter() (short-circuits) then reads the shared task set, which must be narrowed
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner should have entered resetTasks")
+        .isTrue();
+
+    loser.start();
+    // release the winner once the loser has blocked on the monitor or finished its stale read
+    awaitLoserSettled(loser, observedTaskCount);
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as(
+            "the losing UNION branch must observe the narrowed single-file task set, never the "
+                + "full %s-file set (which would rewrite every file and duplicate rows, issue "
+                + "#18004)",
+            FILE_COUNT)
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  // waits until the loser can progress no further before releasing the winner: either blocked on
+  // the scan monitor (synchronized filter()) or already done recording its task count
+  private static void awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return;
+      }
+      Thread.sleep(1);
+    }
+  }

Review Comment:
   Could we add an overall `@Timeout`, fail if either coordination wait times out, and assert that both threads terminate after the timed joins? Otherwise, the test can pass without establishing the required interleaving. This also follows the existing Iceberg concurrency-test pattern.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4198342695


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.Timeout;
+
+/** Tests concurrent runtime filtering on a shared copy-on-write scan. */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 2;
+  private static final long TIMEOUT_SECONDS = 10;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery unions two branches over one shared scan, filtered concurrently
+  // under AQE; the losing branch must see the narrowed task set, not the full one.
+  @TestTemplate
+  @Timeout(value = 60, unit = TimeUnit.SECONDS)
+  public void concurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner did not enter resetTasks")
+        .isTrue();
+
+    loser.start();
+    assertThat(awaitLoserSettled(loser, observedTaskCount))
+        .as("loser did not block on the scan monitor or record a result")
+        .isTrue();
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+    assertThat(winner.isAlive()).as("winner did not terminate").isFalse();
+    assertThat(loser.isAlive()).as("loser did not terminate").isFalse();
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as("losing branch must observe the narrowed single-file task set, not the full set")
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  private static boolean awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return true;
+      }
+      Thread.sleep(1);
+    }
+    return false;
+  }
+
+  // mirrors SparkScanBuilder#buildCopyOnWriteScan but returns an instrumented subclass

Review Comment:
   Removed this one as well



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java:
##########
@@ -124,7 +124,8 @@ public NamedReference[] filterAttributes() {
   }
 
   @Override
-  public void filter(Predicate[] predicates) {
+  // serialize concurrent filter() calls on a shared scan

Review Comment:
   yes, removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4019776458


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */

Review Comment:
   Could we avoid tying the class description to a specific issue? Tests should describe the behavior they cover rather than the history that prompted them. For example: `Tests concurrent runtime filtering on a shared copy-on-write scan.`



##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;

Review Comment:
   Could we reduce this to 2? The test only needs to distinguish the full task set from the one-file filtered set, and creating ten files requires ten separate inserts in every test run.



##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery becomes a UNION whose branches share one scan, which Spark can
+  // filter() concurrently under AQE. Without synchronization a branch reads the full task set and
+  // rewrites every file, duplicating rows (#18004).
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {

Review Comment:
   Could we drop the `test` prefix and name this `concurrentRuntimeFilteringNarrowsSharedScan`? New tests use descriptive method names without a `test` prefix.



##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Regression test for the copy-on-write runtime-filter concurrency bug (issue #18004). */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 10;
+  private static final long TIMEOUT_SECONDS = 30;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery becomes a UNION whose branches share one scan, which Spark can
+  // filter() concurrently under AQE. Without synchronization a branch reads the full task set and
+  // rewrites every file, duplicating rows (#18004).
+  @TestTemplate
+  public void testConcurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    // one single-row file per INSERT, so the narrowed set (1) is clearly distinct from the full set
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    // memoize the full task set and groups exactly as query planning would
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    // `_file IN (<one location>)` runtime predicate, matching what Spark passes for a COW filter
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    // winner: narrows the scan, then parks in resetTasks while holding the scan monitor
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    // loser: runs filter() (short-circuits) then reads the shared task set, which must be narrowed
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner should have entered resetTasks")
+        .isTrue();
+
+    loser.start();
+    // release the winner once the loser has blocked on the monitor or finished its stale read
+    awaitLoserSettled(loser, observedTaskCount);
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as(
+            "the losing UNION branch must observe the narrowed single-file task set, never the "
+                + "full %s-file set (which would rewrite every file and duplicate rows, issue "
+                + "#18004)",
+            FILE_COUNT)
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  // waits until the loser can progress no further before releasing the winner: either blocked on
+  // the scan monitor (synchronized filter()) or already done recording its task count
+  private static void awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return;
+      }
+      Thread.sleep(1);
+    }
+  }

Review Comment:
   Could we fail when this deadline is reached instead of silently continuing? If the loser is not scheduled in time, releasing the winner can let the test pass even after `synchronized` is removed. Please also check the boolean result of `releaseWinner.await` for the same reason.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r4138649949


##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.Timeout;
+
+/** Tests concurrent runtime filtering on a shared copy-on-write scan. */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 2;
+  private static final long TIMEOUT_SECONDS = 10;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery unions two branches over one shared scan, filtered concurrently
+  // under AQE; the losing branch must see the narrowed task set, not the full one.
+  @TestTemplate
+  @Timeout(value = 60, unit = TimeUnit.SECONDS)
+  public void concurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner did not enter resetTasks")
+        .isTrue();
+
+    loser.start();
+    assertThat(awaitLoserSettled(loser, observedTaskCount))
+        .as("loser did not block on the scan monitor or record a result")
+        .isTrue();
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+    assertThat(winner.isAlive()).as("winner did not terminate").isFalse();
+    assertThat(loser.isAlive()).as("loser did not terminate").isFalse();
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as("losing branch must observe the narrowed single-file task set, not the full set")
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  private static boolean awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return true;
+      }
+      Thread.sleep(1);
+    }
+    return false;
+  }
+
+  // mirrors SparkScanBuilder#buildCopyOnWriteScan but returns an instrumented subclass

Review Comment:
   id remove these comments as well.  
   
   imo, they seem:
   * AI generated 
   * hard to maintain)
   * just for a test class (low value)
   * fairly obvious.



##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCopyOnWriteScanConcurrency.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
+import org.apache.iceberg.BatchScan;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MetadataColumns;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.metrics.ScanReport;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkReadConf;
+import org.apache.iceberg.spark.TestBaseWithCatalog;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.Timeout;
+
+/** Tests concurrent runtime filtering on a shared copy-on-write scan. */
+public class TestSparkCopyOnWriteScanConcurrency extends TestBaseWithCatalog {
+
+  private static final int FILE_COUNT = 2;
+  private static final long TIMEOUT_SECONDS = 10;
+  private static final long JOIN_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(TIMEOUT_SECONDS);
+
+  @AfterEach
+  public void removeTables() {
+    sql("DROP TABLE IF EXISTS %s", tableName);
+  }
+
+  // A COW UPDATE with a subquery unions two branches over one shared scan, filtered concurrently
+  // under AQE; the losing branch must see the narrowed task set, not the full one.
+  @TestTemplate
+  @Timeout(value = 60, unit = TimeUnit.SECONDS)
+  public void concurrentRuntimeFilteringNarrowsSharedScan() throws Exception {
+    sql(
+        "CREATE TABLE %s (id BIGINT, data STRING) USING iceberg "
+            + "TBLPROPERTIES ('write.update.mode'='copy-on-write')",
+        tableName);
+
+    for (int i = 0; i < FILE_COUNT; i++) {
+      sql("INSERT INTO %s VALUES (%d, 'data-%d')", tableName, i, i);
+    }
+
+    Table table = validationCatalog.loadTable(tableIdent);
+
+    CountDownLatch enteredResetTasks = new CountDownLatch(1);
+    CountDownLatch releaseWinner = new CountDownLatch(1);
+    InstrumentedCopyOnWriteScan scan = newInstrumentedScan(table, enteredResetTasks, releaseWinner);
+
+    List<FileScanTask> plannedTasks = scan.tasks();
+    scan.taskGroups();
+    assertThat(plannedTasks).as("expected one task per single-row file").hasSize(FILE_COUNT);
+
+    String targetLocation = plannedTasks.get(0).file().location();
+    Predicate[] predicates = {filePathInPredicate(targetLocation)};
+
+    AtomicReference<Throwable> winnerFailure = new AtomicReference<>();
+    AtomicReference<Throwable> loserFailure = new AtomicReference<>();
+    AtomicInteger observedTaskCount = new AtomicInteger(-1);
+    AtomicInteger observedTaskGroupFileCount = new AtomicInteger(-1);
+
+    Thread winner =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+              } catch (Throwable t) {
+                winnerFailure.set(t);
+              }
+            },
+            "cow-filter-winner");
+
+    Thread loser =
+        new Thread(
+            () -> {
+              try {
+                scan.filter(predicates);
+                observedTaskCount.set(scan.tasks().size());
+                observedTaskGroupFileCount.set(countFiles(scan.taskGroups()));
+              } catch (Throwable t) {
+                loserFailure.set(t);
+              }
+            },
+            "cow-filter-loser");
+
+    winner.start();
+    assertThat(enteredResetTasks.await(TIMEOUT_SECONDS, TimeUnit.SECONDS))
+        .as("winner did not enter resetTasks")
+        .isTrue();
+
+    loser.start();
+    assertThat(awaitLoserSettled(loser, observedTaskCount))
+        .as("loser did not block on the scan monitor or record a result")
+        .isTrue();
+    releaseWinner.countDown();
+
+    winner.join(JOIN_TIMEOUT_MILLIS);
+    loser.join(JOIN_TIMEOUT_MILLIS);
+    assertThat(winner.isAlive()).as("winner did not terminate").isFalse();
+    assertThat(loser.isAlive()).as("loser did not terminate").isFalse();
+
+    if (winnerFailure.get() != null) {
+      throw new AssertionError("winner branch failed", winnerFailure.get());
+    }
+    if (loserFailure.get() != null) {
+      throw new AssertionError("loser branch failed", loserFailure.get());
+    }
+
+    assertThat(observedTaskCount.get())
+        .as("losing branch must observe the narrowed single-file task set, not the full set")
+        .isEqualTo(1);
+    assertThat(observedTaskGroupFileCount.get())
+        .as("task groups must reflect the narrowed single-file set")
+        .isEqualTo(1);
+  }
+
+  private static boolean awaitLoserSettled(Thread loser, AtomicInteger observedTaskCount)
+      throws InterruptedException {
+    long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS);
+    while (System.nanoTime() < deadlineNanos) {
+      Thread.State state = loser.getState();
+      if (state == Thread.State.BLOCKED
+          || state == Thread.State.TERMINATED
+          || observedTaskCount.get() != -1) {
+        return true;
+      }
+      Thread.sleep(1);
+    }
+    return false;
+  }
+
+  // mirrors SparkScanBuilder#buildCopyOnWriteScan but returns an instrumented subclass

Review Comment:
   same, remove the comment



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java:
##########
@@ -124,7 +124,8 @@ public NamedReference[] filterAttributes() {
   }
 
   @Override
-  public void filter(Predicate[] predicates) {
+  // serialize concurrent filter() calls on a shared scan

Review Comment:
   there's no need for the comment here.. its describing synchronzied keyword



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#issuecomment-5574519292

   relates to https://github.com/apache/iceberg/issues/18004


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#discussion_r3995541111


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java:
##########
@@ -124,7 +124,10 @@ public NamedReference[] filterAttributes() {
   }
 
   @Override
-  public void filter(Predicate[] predicates) {
+  // synchronized on the same monitor as tasks()/taskGroups()/resetTasks() so the non-atomic

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Fix concurrent runtime filtering on shared scan for COW UPDATE with subquery [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #18005:
URL: https://github.com/apache/iceberg/pull/18005#issuecomment-6051146231

   Thanks @hemanthboyina !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org