You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2024/01/26 20:39:26 UTC

[PR] Spark 3.5: Support executor cache locality [iceberg]

aokolnychyi opened a new pull request, #9563:
URL: https://github.com/apache/iceberg/pull/9563

   This PR adds an ability to enable executor cache locality. The new SQL property is off by default as it does not make sense in all cases and may introduce unnecessary task waits.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1475438902


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java:
##########
@@ -331,4 +331,24 @@ private long driverMaxResultSize() {
     SparkConf sparkConf = spark.sparkContext().conf();
     return sparkConf.getSizeAsBytes(DRIVER_MAX_RESULT_SIZE, DRIVER_MAX_RESULT_SIZE_DEFAULT);
   }
+
+  public boolean executorCacheLocalityEnabled() {
+    return executorCacheEnabled() && executorCacheLocalityEnabledInternal();
+  }
+
+  private boolean executorCacheEnabled() {
+    return confParser
+        .booleanConf()
+        .sessionConf(SparkSQLProperties.EXECUTOR_CACHE_ENABLED)
+        .defaultValue(SparkSQLProperties.EXECUTOR_CACHE_ENABLED_DEFAULT)
+        .parse();
+  }
+
+  private boolean executorCacheLocalityEnabledInternal() {

Review Comment:
   > My original approach was to enable the executor cache locality by default only if dynamic allocation is disabled.
   
   That's my first thought too. Then I realize what if users want to enable this anyway. It should be up to the users to decide. 
   
   What about log a warning when both dynamic allocation and executor cache locality are enabled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474861017


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;
+  }
+
+  private static int hash(StructLike struct) {

Review Comment:
   If there is a cheap way to leverage `JavaHashes`, I am all for it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474860162


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;
+  }
+
+  private static int hash(StructLike struct) {

Review Comment:
   This logic would cause issues if the underlying `StructLike` instances may have different value representations for the same column. For instance, one struct has `String` and the other one has some other form of `CharSequence`. I am not sure how realistic that would be. Even if that happens, there would be no correctness issues as such tasks would simply be assigned to different slots.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474847017


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java:
##########
@@ -331,4 +331,24 @@ private long driverMaxResultSize() {
     SparkConf sparkConf = spark.sparkContext().conf();
     return sparkConf.getSizeAsBytes(DRIVER_MAX_RESULT_SIZE, DRIVER_MAX_RESULT_SIZE_DEFAULT);
   }
+
+  public boolean executorCacheLocalityEnabled() {
+    return executorCacheEnabled() && executorCacheLocalityEnabledInternal();
+  }
+
+  private boolean executorCacheEnabled() {
+    return confParser
+        .booleanConf()
+        .sessionConf(SparkSQLProperties.EXECUTOR_CACHE_ENABLED)
+        .defaultValue(SparkSQLProperties.EXECUTOR_CACHE_ENABLED_DEFAULT)
+        .parse();
+  }
+
+  private boolean executorCacheLocalityEnabledInternal() {

Review Comment:
   My original approach was to enable the executor cache locality by default only if dynamic allocation is disabled. After thinking more about it, I decided to simply disable it by default no matter whether static or dynamic allocation is used. As of right now, folks have to opt-in explicitly to enable executor cache locality. That way, we ensure there are no extra waits added on our end as we can't guarantee the locality would be beneficial.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474842991


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;
+  }
+
+  private static int hash(StructLike struct) {

Review Comment:
   It would require having access to `task.spec()` to know the struct type, which I think is [expensive to get](https://github.com/apache/iceberg/pull/9563#discussion_r1474841847).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1476795233


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   It looks like the schema and spec caches are doing their jobs, I don't see much time spent on that, I'll switch to using specs. Good call, @advancedxy!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1476795233


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   It looks like the schema and spec caches are doing their jobs, I don't see much time spent on that, I'll switch to using specs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1475440771


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadDelete.java:
##########
@@ -85,6 +86,30 @@ public static void clearTestSparkCatalogCache() {
     TestSparkCatalog.clearTables();
   }
 
+  @Test
+  public void testDeleteWithExecutorCacheLocality() throws NoSuchTableException {
+    createAndInitPartitionedTable();
+
+    append(tableName, new Employee(1, "hr"), new Employee(2, "hr"));
+    append(tableName, new Employee(3, "hr"), new Employee(4, "hr"));
+    append(tableName, new Employee(1, "hardware"), new Employee(2, "hardware"));
+    append(tableName, new Employee(3, "hardware"), new Employee(4, "hardware"));
+
+    createBranchIfNeeded();
+
+    withSQLConf(
+        ImmutableMap.of(SparkSQLProperties.EXECUTOR_CACHE_LOCALITY_ENABLED, "true"),
+        () -> {
+          sql("DELETE FROM %s WHERE id = 1", commitTarget());
+          sql("DELETE FROM %s WHERE id = 3", commitTarget());
+
+          assertEquals(

Review Comment:
   Ah, I see. Iceberg's spark tests only support `local` mode.  There's a `local-cluster` mode which requires extra setups. I think it's fine to left it as it is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#issuecomment-1927759989

   Thanks, @advancedxy @rdblue! I am going to test this with our RC on a cluster. I can't cover everything locally. I tested the initial prototype on a cluster and it worked well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474613749


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   how about checks `task.spec().isPartitioned()` instead? Otherwise, `VoidTransform` in V1 table is not handled in the above code.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java:
##########
@@ -331,4 +331,24 @@ private long driverMaxResultSize() {
     SparkConf sparkConf = spark.sparkContext().conf();
     return sparkConf.getSizeAsBytes(DRIVER_MAX_RESULT_SIZE, DRIVER_MAX_RESULT_SIZE_DEFAULT);
   }
+
+  public boolean executorCacheLocalityEnabled() {
+    return executorCacheEnabled() && executorCacheLocalityEnabledInternal();
+  }
+
+  private boolean executorCacheEnabled() {
+    return confParser
+        .booleanConf()
+        .sessionConf(SparkSQLProperties.EXECUTOR_CACHE_ENABLED)
+        .defaultValue(SparkSQLProperties.EXECUTOR_CACHE_ENABLED_DEFAULT)
+        .parse();
+  }
+
+  private boolean executorCacheLocalityEnabledInternal() {

Review Comment:
   I'm a little concerned that this doesn't play well with Spark's `dynamicAllocation` which should be enabled by default for most production systems.
   
   Did you test how would this work with dynamic allocation enabled?



##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadDelete.java:
##########
@@ -85,6 +86,30 @@ public static void clearTestSparkCatalogCache() {
     TestSparkCatalog.clearTables();
   }
 
+  @Test
+  public void testDeleteWithExecutorCacheLocality() throws NoSuchTableException {
+    createAndInitPartitionedTable();
+
+    append(tableName, new Employee(1, "hr"), new Employee(2, "hr"));
+    append(tableName, new Employee(3, "hr"), new Employee(4, "hr"));
+    append(tableName, new Employee(1, "hardware"), new Employee(2, "hardware"));
+    append(tableName, new Employee(3, "hardware"), new Employee(4, "hardware"));
+
+    createBranchIfNeeded();
+
+    withSQLConf(
+        ImmutableMap.of(SparkSQLProperties.EXECUTOR_CACHE_LOCALITY_ENABLED, "true"),
+        () -> {
+          sql("DELETE FROM %s WHERE id = 1", commitTarget());
+          sql("DELETE FROM %s WHERE id = 3", commitTarget());
+
+          assertEquals(

Review Comment:
   Seems it only checks for records equality, but doesn't check the executor cache locality?
   
   I think we may check spark RDD's ` getPreferredLocations` instead?



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;
+  }
+
+  private static int hash(StructLike struct) {

Review Comment:
   is it possible to reuse `org.apache.iceberg.types.JavaHash`(es) here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474842991


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;
+  }
+
+  private static int hash(StructLike struct) {

Review Comment:
   It would require having access to `task.spec()`, which I think is [expensive to get](https://github.com/apache/iceberg/pull/9563#discussion_r1474841847).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1476443719


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   I think you are right. Let me profile that part and see how expensive it is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1476899638


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java:
##########
@@ -331,4 +331,24 @@ private long driverMaxResultSize() {
     SparkConf sparkConf = spark.sparkContext().conf();
     return sparkConf.getSizeAsBytes(DRIVER_MAX_RESULT_SIZE, DRIVER_MAX_RESULT_SIZE_DEFAULT);
   }
+
+  public boolean executorCacheLocalityEnabled() {
+    return executorCacheEnabled() && executorCacheLocalityEnabledInternal();
+  }
+
+  private boolean executorCacheEnabled() {
+    return confParser
+        .booleanConf()
+        .sessionConf(SparkSQLProperties.EXECUTOR_CACHE_ENABLED)
+        .defaultValue(SparkSQLProperties.EXECUTOR_CACHE_ENABLED_DEFAULT)
+        .parse();
+  }
+
+  private boolean executorCacheLocalityEnabledInternal() {

Review Comment:
   My worry is that we don't really know if enabling this with dynamic allocation is going to hurt. For instance, it still may make sense if the min number of executors is big enough or if the cluster is hot. Given that we would also have to add logic to parse the dynamic allocation config, I'd probably not log it and trust the person setting this for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1476899638


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java:
##########
@@ -331,4 +331,24 @@ private long driverMaxResultSize() {
     SparkConf sparkConf = spark.sparkContext().conf();
     return sparkConf.getSizeAsBytes(DRIVER_MAX_RESULT_SIZE, DRIVER_MAX_RESULT_SIZE_DEFAULT);
   }
+
+  public boolean executorCacheLocalityEnabled() {
+    return executorCacheEnabled() && executorCacheLocalityEnabledInternal();
+  }
+
+  private boolean executorCacheEnabled() {
+    return confParser
+        .booleanConf()
+        .sessionConf(SparkSQLProperties.EXECUTOR_CACHE_ENABLED)
+        .defaultValue(SparkSQLProperties.EXECUTOR_CACHE_ENABLED_DEFAULT)
+        .parse();
+  }
+
+  private boolean executorCacheLocalityEnabledInternal() {

Review Comment:
   My worry is that we don't really know if enabling this with dynamic allocation is going to hurt. For instance, it still may make sense if the min number of executors is big enough or if the cluster is hot. Given that we would also have to add logic to parse the dynamic allocation config, I'd probably not log it and trust the person setting this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474841847


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   That's what I started with but `task.spec()` may be quite expensive.
   
   ```
   @Override
   public PartitionSpec spec() {
     if (spec == null) {
       synchronized (this) {
         if (spec == null) {
           this.spec = PartitionSpecParser.fromJson(schema(), specString);
         }
       }
     }
     return spec;
   }
   ```
   
   Even though there is a cache in `PartitionSpecParser`, we need to do to a lookup by the spec json. I don't think void transforms will be an issue as we check for presence of deletes and V1 tables can't have them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1474850481


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadDelete.java:
##########
@@ -85,6 +86,30 @@ public static void clearTestSparkCatalogCache() {
     TestSparkCatalog.clearTables();
   }
 
+  @Test
+  public void testDeleteWithExecutorCacheLocality() throws NoSuchTableException {
+    createAndInitPartitionedTable();
+
+    append(tableName, new Employee(1, "hr"), new Employee(2, "hr"));
+    append(tableName, new Employee(3, "hr"), new Employee(4, "hr"));
+    append(tableName, new Employee(1, "hardware"), new Employee(2, "hardware"));
+    append(tableName, new Employee(3, "hardware"), new Employee(4, "hardware"));
+
+    createBranchIfNeeded();
+
+    withSQLConf(
+        ImmutableMap.of(SparkSQLProperties.EXECUTOR_CACHE_LOCALITY_ENABLED, "true"),
+        () -> {
+          sql("DELETE FROM %s WHERE id = 1", commitTarget());
+          sql("DELETE FROM %s WHERE id = 3", commitTarget());
+
+          assertEquals(

Review Comment:
   I am afraid we can't really test this as `SparkUtil$executorLocations` would return an empty list in our local testing env. This test is to simply ensure nothing breaks if run on the driver.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1475446827


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   > That's what I started with but task.spec() may be quite expensive.
   
   I think we already called `specs()` in `SparkPartitioningAwareScan` to preserve data grouping. If it's quite expensive to call `task.spec()`, it would be better that we can reduce this overhead. 
   
   First thing came out of my mind: how about add a `int specId()` method in the  `PartitionScanTask` interface? The specId should be easier to store and retrieve.  In the driver side, we can leverage `table.specs()` to retrieve the actual spec.
   
   > I don't think void transforms will be an issue as we check for presence of deletes and V1 tables can't have them.
   
   Maybe some extreme case, such as: v1 table with `identity(c1)` -- dropped partition transform -->  v1 table with void transform -- upgrade to v2 --> v2 table with void transform --> added deletes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9563:
URL: https://github.com/apache/iceberg/pull/9563#discussion_r1476914683


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPlanningUtil.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionScanTask;
+import org.apache.iceberg.ScanTask;
+import org.apache.iceberg.ScanTaskGroup;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.hadoop.Util;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.Tasks;
+import org.apache.iceberg.util.ThreadPools;
+
+class SparkPlanningUtil {
+
+  public static final String[] NO_LOCATION_PREFERENCE = new String[0];
+
+  private SparkPlanningUtil() {}
+
+  public static String[][] fetchBlockLocations(
+      FileIO io, List<? extends ScanTaskGroup<?>> taskGroups) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    Tasks.range(taskGroups.size())
+        .stopOnFailure()
+        .executeWith(ThreadPools.getWorkerPool())
+        .run(index -> locations[index] = Util.blockLocations(io, taskGroups.get(index)));
+
+    return locations;
+  }
+
+  public static String[][] assignExecutors(
+      List<? extends ScanTaskGroup<?>> taskGroups, List<String> executorLocations) {
+    String[][] locations = new String[taskGroups.size()][];
+
+    for (int index = 0; index < taskGroups.size(); index++) {
+      locations[index] = assign(taskGroups.get(index), executorLocations);
+    }
+
+    return locations;
+  }
+
+  private static String[] assign(ScanTaskGroup<?> taskGroup, List<String> executorLocations) {
+    List<String> locations = Lists.newArrayList();
+
+    for (ScanTask task : taskGroup.tasks()) {
+      if (task.isFileScanTask()) {
+        FileScanTask fileTask = task.asFileScanTask();
+        if (isPartitioned(fileTask) && !fileTask.deletes().isEmpty()) {
+          int index = Math.floorMod(hash(fileTask.partition()), executorLocations.size());
+          String executorLocation = executorLocations.get(index);
+          locations.add(executorLocation);
+        }
+      }
+    }
+
+    return locations.toArray(NO_LOCATION_PREFERENCE);
+  }
+
+  private static boolean isPartitioned(PartitionScanTask task) {
+    return task.partition() != null && task.partition().size() > 0;

Review Comment:
   I switched to proper hashes, will add another JMH benchmark later but seems to perform alright.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi merged PR #9563:
URL: https://github.com/apache/iceberg/pull/9563


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org