You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2023/07/26 22:43:46 UTC
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #7714: Spark 3.4: Adaptive split size
aokolnychyi commented on code in PR #7714:
URL: https://github.com/apache/iceberg/pull/7714#discussion_r1275552332
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -232,6 +232,16 @@ protected synchronized List<ScanTaskGroup<T>> taskGroups() {
return taskGroups;
}
+ private long targetSplitSize() {
+ if (readConf().adaptiveSplitSizeEnabled()) {
+ long scanSize = tasks().stream().mapToLong(ScanTask::sizeBytes).sum();
+ int parallelism = sparkContext().defaultParallelism();
Review Comment:
The default parallelism is populated via `TaskScheduler` from `SchedulerBackend`:
```
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}
```
The core count is being updated each time an executor is added/dropped so dynamic execution should work.
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -232,6 +232,16 @@ protected synchronized List<ScanTaskGroup<T>> taskGroups() {
return taskGroups;
}
+ private long targetSplitSize() {
+ if (readConf().adaptiveSplitSizeEnabled()) {
+ long scanSize = tasks().stream().mapToLong(ScanTask::sizeBytes).sum();
+ int parallelism = sparkContext().defaultParallelism();
Review Comment:
The default parallelism is populated via `TaskScheduler` from `SchedulerBackend`:
```
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}
```
The core count is being updated each time an executor is added/dropped so dynamic allocation should work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org