You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "davseitsev (via GitHub)" <gi...@apache.org> on 2024/11/25 12:13:31 UTC
[I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
davseitsev opened a new issue, #11648:
URL: https://github.com/apache/iceberg/issues/11648
### Apache Iceberg version
1.7.0 (latest release)
### Query engine
Spark
### Please describe the bug š
We have a scheduled Spark maintenance jobs which runs all necessary Spark actions to keep our data lake clean and healthy. Recently we have upgraded iceberg from 1.1.0 to 1.7.0 and enabled deletes compaction.
We noticed that data compaction of tables with position deletes is really slow and it blocks almost all other data compaction jobs on the cluster. And here slow means it almost never ends, you can wait a few hours and se zero progress.
Investigation showed that there are some threads on Spark workers which consumes all the CPU called `iceberg-delete-worker-pool-%`. Here is flame graph:
<img width="1308" alt="image" src="https://github.com/user-attachments/assets/03dc1c61-2773-4699-b43b-a0f8e280cbe1">
Thread dump example:
```
"iceberg-delete-worker-pool-0" #190 daemon prio=5 os_prio=0 cpu=1108299.90ms elapsed=7369.57s tid=0x0000aaaaed08d370 nid=0x37a0 runnable [0x0000ffff2a0da000]
java.lang.Thread.State: RUNNABLE
at org.apache.iceberg.types.JavaHashes.hashCode(JavaHashes.java:34)
at org.apache.iceberg.util.CharSequenceWrapper.hashCode(CharSequenceWrapper.java:85)
at java.util.HashMap.hash(java.base@17.0.13/HashMap.java:338)
at java.util.HashMap.getNode(java.base@17.0.13/HashMap.java:568)
at java.util.HashMap.get(java.base@17.0.13/HashMap.java:556)
at org.apache.iceberg.util.CharSequenceMap.get(CharSequenceMap.java:92)
at java.util.Map.computeIfAbsent(java.base@17.0.13/Map.java:1052)
at org.apache.iceberg.deletes.Deletes.toPositionIndexes(Deletes.java:152)
at org.apache.iceberg.data.BaseDeleteLoader.readPosDeletes(BaseDeleteLoader.java:169)
at org.apache.iceberg.data.BaseDeleteLoader.lambda$getOrReadPosDeletes$3(BaseDeleteLoader.java:160)
at org.apache.iceberg.data.BaseDeleteLoader$$Lambda$3227/0x0000009002085c10.get(Unknown Source)
at org.apache.iceberg.spark.SparkExecutorCache.lambda$loadFunc$0(SparkExecutorCache.java:122)
at org.apache.iceberg.spark.SparkExecutorCache$$Lambda$3230/0x000000900208f190.apply(Unknown Source)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache$$Lambda$3231/0x000000900208f3d8.apply(Unknown Source)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$2184/0x0000009001d62478.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.compute(java.base@17.0.13/ConcurrentHashMap.java:1916)
- locked <0x000000068b0009a0> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at org.apache.iceberg.spark.SparkExecutorCache.getOrLoad(SparkExecutorCache.java:114)
at org.apache.iceberg.spark.source.BaseReader$SparkDeleteFilter$CachingDeleteLoader.getOrLoad(BaseReader.java:297)
at org.apache.iceberg.data.BaseDeleteLoader.getOrReadPosDeletes(BaseDeleteLoader.java:160)
at org.apache.iceberg.data.BaseDeleteLoader.lambda$loadPositionDeletes$2(BaseDeleteLoader.java:150)
at org.apache.iceberg.data.BaseDeleteLoader$$Lambda$3224/0x0000009002084498.apply(Unknown Source)
at org.apache.iceberg.data.BaseDeleteLoader.lambda$execute$7(BaseDeleteLoader.java:236)
at org.apache.iceberg.data.BaseDeleteLoader$$Lambda$3226/0x0000009002085408.run(Unknown Source)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.13/Executors.java:539)
at java.util.concurrent.FutureTask.run(java.base@17.0.13/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.13/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.13/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.13/Thread.java:840)
```
After some research I figured out that there is a cache of delete files which loads the whole delete file even if you need position deletes only for single data file. As far as I understand in our case the size of the cache is really small and it's constantly evicted. It dramatically slows down reading deletes. When I turned it off `spark.sql.iceberg.executor-cache.enabled=false`, jobs which ran for a few hours without progress started to finish in about 1 minute.
I don't see any benefit from having this cache for RewriteDataFilesSparkAction and I suggest to disable it by default for this action.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-4238392610
Hello @aokolnychyi , @anuragmantri , could you please take a look at this issue once again? `RewriteDataFilesSparkAction` was significantly slowed down by the disabled executor cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-2501506598
@davseitsev can you please also try setting to true ?
```
spark.sql.iceberg.executor-cache.locality.enabled
```
please ref : https://github.com/apache/iceberg/pull/9563#issue-2102869774
Also for this :
> As far as I understand in our case the size of the cache is really small and it's constantly evicted
we can increase this via :
https://github.com/apache/iceberg/blob/f3560871564c95a0e7b2bff3ca6ecb2e08726d01/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java#L86
Do you have memory profiles / heap dump for it ? 128 MB is decent, what I am a bit skeptical of is it's just that delete from diff partitions landed in same executor so above might help as to process it, as cache hit would minimized.
There are some gotchas for it obviously for ex : https://github.com/apache/iceberg/pull/8755#issuecomment-1754135275
as @nastra recommended good discussion on mailing chain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5441068853
@kinolaev, sorry I was late to get back on this, I will take a look at your analysis and get back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-4857849728
I wrote a simple benchmark to compare the performance of `Deletes.toPositionIndexes` between the current version and the version in my PR:
```java
@Fork(1)
@State(Scope.Benchmark)
@Warmup(iterations = 3)
@Measurement(iterations = 5)
@BenchmarkMode(Mode.SingleShotTime)
@Timeout(time = 10, timeUnit = TimeUnit.MINUTES)
public class ToPositionIndexesBenchmark {
@Param({"1000000"})
private int numDeletes;
@Param({"1", "10", "100", "1000", "10000", "100000", "1000000"})
private int numDataFiles;
private List<StructLike> deletes;
private String pathTemplate =
"s3://pretty-long-bucket-name/pretty-long-namespace-name/pretty-long-table-name/data/%07d-data.parquet";
@Setup
public void setupBenchmark() {
deletes = Lists.newArrayListWithExpectedSize(numDeletes);
int numDeletesPerDataFile = numDeletes / numDataFiles;
for (int index = 0; index < numDeletes; index++) {
String path = String.format(Locale.ROOT, pathTemplate, index / numDeletesPerDataFile);
deletes.add(new PositionDeleteRow(path, index % numDeletesPerDataFile));
}
}
@Benchmark
@Threads(1)
public void current(Blackhole blackhole) {
blackhole.consume(Deletes.toPositionIndexesMain(CloseableIterable.withNoopClose(deletes)));
}
@Benchmark
@Threads(1)
public void pr15714(Blackhole blackhole) {
blackhole.consume(Deletes.toPositionIndexes(CloseableIterable.withNoopClose(deletes)));
}
// ...
}
```
Here are the results:
```
Benchmark (numDataFiles) (numDeletes) Mode Cnt Score Error Units
ToPositionIndexesBenchmark.current 1 1000000 ss 5 0.315 Ā± 0.017 s/op
ToPositionIndexesBenchmark.current 10 1000000 ss 5 0.322 Ā± 0.047 s/op
ToPositionIndexesBenchmark.current 100 1000000 ss 5 0.323 Ā± 0.035 s/op
ToPositionIndexesBenchmark.current 1000 1000000 ss 5 0.135 Ā± 0.003 s/op
ToPositionIndexesBenchmark.current 10000 1000000 ss 5 0.326 Ā± 0.016 s/op
ToPositionIndexesBenchmark.current 100000 1000000 ss 5 0.186 Ā± 0.060 s/op
ToPositionIndexesBenchmark.current 1000000 1000000 ss 5 0.803 Ā± 0.127 s/op
ToPositionIndexesBenchmark.pr15714 1 1000000 ss 5 0.033 Ā± 0.043 s/op
ToPositionIndexesBenchmark.pr15714 10 1000000 ss 5 0.034 Ā± 0.042 s/op
ToPositionIndexesBenchmark.pr15714 100 1000000 ss 5 0.025 Ā± 0.047 s/op
ToPositionIndexesBenchmark.pr15714 1000 1000000 ss 5 0.026 Ā± 0.047 s/op
ToPositionIndexesBenchmark.pr15714 10000 1000000 ss 5 0.050 Ā± 0.001 s/op
ToPositionIndexesBenchmark.pr15714 100000 1000000 ss 5 0.058 Ā± 0.007 s/op
ToPositionIndexesBenchmark.pr15714 1000000 1000000 ss 5 0.605 Ā± 0.159 s/op
```
When a delete file contains only one position per data file, the updated version performs at the same level. Otherwise, the updated version is 3 to 12 times faster, depending on the ratio of positions per data file.
If this improvement is still not sufficient, we should consider skipping the cache for position deletes entirely, just as is already done for DVs. Additionally, it's worth to noting that the cache affects not only rewrite operations but all read operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456733244
No more questions about it then :)
@anuragmantri, will you be able to create a PR with the `cache-delete-files` parameter before 1.12.0? If not, I could do it tomorrow. Please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456082806
I read the code and all the comments here and I agree with Russell. We probably don't need to revert the ability to disable executor cache, but we could relax it to be overridable. I will create a PR for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5441087719
Are you using mostly partition scoped position delete files? I believe we were (even at that point in time last year) moving to the general philosophy that using partition scoped deletes was generally a bad idea. I don't think there is an issue with allowing the cache to be optionally turned on but at the moment it would only really apply partition delete files in V2 tables right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-2497946248
@davseitsev I think this is a good topic to bring up on the Dev mailing list to reach a broader audience and facilitate a discussion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-4103763216
I'd like to reopen the issue. I enabled executor cache in RewriteDataFilesSparkAction and it didn't cause any slowness or a hang. Instead it saved me millions of requests to an s3 bucket. Without cache spark loads every delete files for every data file. For example, if a table has data files with id bounds [11,20] and [20,30] and a delete file with id bounds [15,25] spark will load the delete file twice. In my case if I accidentally skip daily maintenance for several days, spark will make hundreds of millions unnecessary requests to s3.
Here is my cache stats for 3 days after 3 days without maintenance:
hitCount=219680918, loadSuccessCount=434508
hitCount=18618032, loadSuccessCount=106231
hitCount=4895082, loadSuccessCount=24220
That is why I made a PR #15714 that reverts #13820 and #13868. The option to disable cache is untouched in case the problem still exists in some special conditions.
Btw, I found two cases when spark stucks when number of simultaneous connections is limited #15712, #15713. Maybe the problem with cache was only a side effect and not the root cause of slowness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5497116358
Thanks @anuragmantri and @RussellSpitzer, I'm really happy this is finally fixed!
I've extracted the second part - the path rehashing fix - into PR #17864. I'd appreciate it if you could take a look when you have a chance. It's tiny, just 8 lines changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-3100058995
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11648: SparkExecutorCache causes slowness of RewriteDataFilesSparkAction
URL: https://github.com/apache/iceberg/issues/11648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456648167
We should probably turn it off there too, but I think the impacts are much worse on full table scans. Since all of this is V2 I probably wouldn't mess with anything unless someone has a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-2606288708
Unfortunately, we haven't heard back. That said, I may have a guess. I believe it is related to the connection pool we use for reading deletes. The rewrite action submits multiple actions at the same time and they may use the same connection pool.
Overall, I agree we shouldn't be using the executor cache for this action as each partition is rewritten separately. We need a config to disable executor cache for deletes and the action should automatically set it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer closed issue #11648: SparkExecutorCache causes slowness of RewriteDataFilesSparkAction
URL: https://github.com/apache/iceberg/issues/11648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-3185529759
[#12893](https://github.com/apache/iceberg/pull/12893) Does not fix the issue yet, it was a prerequisite, we will disable the cache for rewrite data files spark action in a subsequent PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456344828
Thanks for the clarification - I didn't know the delete granularity default was overridden at the Spark level. Would it make sense to disable the cache for position deletes entirely? I mean for every read, not just compaction. Like for DV. Alternatively, maybe cache usage should depend on the delete granularity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-3152865186
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5440859902
@ebyhr @stevenzwu @RussellSpitzer @huaxingao, you previously approved PRs #13820, #13868, which led to unintended consequences described in the comment above. Please take another look at the issue and my PR #15714, which re-enables the executor cache and improves position deletes caching. Happy to discuss my findings and continue working on the solution with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5441275856
Ah Equality delete files, yeah this doesn't really make sense for position delete files but Equality deletes are a bit of a different story.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5441510246
> this doesn't really make sense for position delete files
@RussellSpitzer, just curious, why wouldn't it make sense? A position delete file contains deletes for many data files, and per my benchmark results, caching the whole file can be almost as fast as a single selective read (default settings + the rehashing issue fixed). With the default target delete file size at 64 MB, position delete files are most probably referenced by multiple data files. This means that whenever a read operation (not only compaction) requires at least two data files referencing the same position delete file, the cache speeds up the operation. Am I missing something? Again - just curious to understand, as I don't currently use position deletes myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-2606336520
I will work on this. I will create a config for disabling executor cache. Thanks for the bug report and inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-4826628564
I believe I've found the root cause of the issue - hash code computation for each row of a position delete file.
When the cache is disabled, `toPositionIndex` pushes the file_path predicate down to a delete file reader, which is the most efficient way to read the relevant rows. When the cache is enabled, the delete file is read fully, and for each row `toPositionIndexes` method calls `CharSequenceMap.computeIfAbsent` that requires a hash computation for the provided `file_path`. As the flame chart shows `computeIfAbsent` method takes at least half of the samples.
The Iceberg spec states that "The rows in the delete file must be sorted by file_path then pos to optimize filtering rows while scanning". That means that we only need to lookup an index when `file_path` is different from the previous row. As far as I know `org.apache.commons.lang3.Strings.CS.equals()` should be cheaper than `org.apache.iceberg.util.CharSequenceWrapper.hashCode()`, so reusing index when `file_path` is the same between contiguous rows should improve performance of `toPositionIndexes` method. I've added the change to my PR (https://github.com/apache/iceberg/pull/15714, commit fa492a6ead4eeb06d0275511b32822c6cb8749f5).
@davseitsev , could you please test if it works for you?
@nastra , could you please reopen the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456552088
We would only ever know at scan time, we can't really rely on the table config and such. My gut action here is still Equality deletes maybe would benefit from the cache. Position Delete users should upgrade to V3 with DVs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5441214403
The main reason I need a delete file cache is that, otherwise, equality delete files are loaded for each referencing data file, causing S3 request counts to skyrocket. I dug into the position delete caching problem simply to make it clear that there is nothing fundamentally wrong with caching position delete files (with default settings, the cache pays off starting from the second lookup, according to my benchmark results above.), and that the root cause of the original issue was just a cache size misconfiguration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-3152877842
This is still active. The PR is pending review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5387856932
I ran a more realistic benchmark with a local S3 server and want to share the results.
**TL;DR**
- **Disabling the cache was the wrong fix.** The real bug is `spark.sql.iceberg.executor-cache.max-total-size` being too small (causing repeated eviction and full re-decode) plus unnecessary per-row rehashing - #15714 fixes the rehashing part (~1.8x faster: ~11.09-11.51s ā ~6.04-6.50s per delete file).
- **Size the cache correctly instead**: in the worked example below (128 data files, a too-small cache), processing time drops from ~25.6 minutes to ~12 seconds.
- **Independent tip:** lowering `write.delete.parquet.row-group-size-bytes` (default 128mb) makes uncached selective reads up to 17x faster, and raises the cache break-even point from ~1 lookup at the 64mb default up to ~18 lookups at 2mb row groups (per the example).
All benchmark code is published in [a separate branch](https://github.com/kinolaev/iceberg/tree/position-delete-benchmarks). The benchmarks were run on a MacBook Pro (M1 Pro CPU, 16gb RAM). The tested position delete file has 67108864 rows (64\*1024\*1024), and its size varies between 63630623 and 66101686 bytes. I picked that size because 64mb is the default value for [write.delete.target-file-size-bytes](https://iceberg.apache.org/docs/latest/configuration/#write-properties).
The time it takes to read deleted positions for a specific data file depends on how many distinct data files the delete file references:
| file_path count | read timing (seconds) |
|---:|---:|
| 4 | 7.706 |
| 8 | 6.909 |
| 16 | 6.325 |
| 32 | 6.206 |
| 64 | 6.181 |
| 128 | 6.296 |
| 256 | 5.792 |
| 512 | 6.091 |
The time for caching doesn't depend on how many data files are referenced; it's 11.09-11.51 seconds on the main branch and 6.04-6.50 seconds on the PR #15714 branch. So, the PR makes the caching process only 1.8 times faster, but that's enough to make it as fast as reading position deletes for just a single data file. In other words, disabling the cache for position deletes is a bad idea.
From what I can tell, the issue has two root causes:
1. `spark.sql.iceberg.executor-cache.max-total-size` was too small
2. hashes were computed for every row in a position delete file
Because the cache was too small, entries got evicted before they could be reused. For example, say you have 128 data files and two 65mb position delete files, each referencing 64 data files. If you're lucky and the first 64 data files you read all happen to be referenced by the same delete file, you'll only need ~12x2 seconds (or ~6x2 with the PR) to process the delete files. But if you're unlucky - the even-numbered data files are referenced by the first delete file and the odd-numbered ones by the second - you'll run into this issue:
1. the first delete file is loaded, ~67 million hashes are computed, and the result `CharSequenceMap<PositionDeleteIndex>` is cached
2. the result of the previous step is evicted, the second delete file is loaded, ~67 million more hashes are computed, and the new result is cached
Then the same steps repeat 64 times. If you have enough RAM for 128 `CharSequenceMap<PositionDeleteIndex>` instances (each 8mb to 3.4gb depending on how sparse the deletes are), you'll probably need only ~12x128=25.6 minutes to process the delete files (or ~6x128=12.8 minutes with the PR). But if you have to gc each map before creating the next one, you get to the point the issue author described as "slow means it almost never ends."
With the PR applied and the cache size set correctly (130mb in this example), processing the delete files would take only ~12 seconds - significantly less than the ~6*128=768 seconds it takes without the cache.
Conclusion: hardcoding `spark.sql.iceberg.executor-cache.max-total-size=false` slowed down compaction both for tables with equality delete files (the data files x delete files request count problem) and for tables with position delete files. The root cause is simply a misconfigured cache size.
@davseitsev, I'd really appreciate it if you could measure `select sum()` (or a similar query that has to apply all the position deletes) on your tables before compaction, with `spark.sql.iceberg.executor-cache.max-total-size=true`, across different cache sizes.
@nastra, @aokolnychyi, @anuragmantri, @singhpk234, please take a look at this issue and PR #15714 - this is critical for anyone compacting tables with equality deletes using Spark.
---
This part is about selective-read performance. It's unrelated to the cache problem, but might be interesting if you want to disable the cache for position delete files. The easy way to improve the performance of reading position delete files is to lower `write.delete.parquet.row-group-size-bytes`:
| file_path count \ row group size | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 4 | 6.490 | 3.514 | 3.507 | 3.896 | 4.366 | 5.458 | 7.706 |
| 8 | 3.274 | 1.808 | 1.949 | 2.284 | 3.577 | 4.539 | 6.909 |
| 16 | 1.664 | 0.986 | 1.134 | 1.637 | 2.201 | 3.586 | 6.325 |
| 32 | 0.809 | 0.563 | 0.750 | 1.435 | 1.817 | 3.364 | 6.206 |
| 64 | 0.389 | 0.361 | 0.545 | 0.935 | 1.696 | 3.202 | 6.181 |
| 128 | 0.221 | 0.256 | 0.464 | 0.959 | 1.756 | 3.533 | 6.296 |
The default value is 128mb, but since `write.delete.target-file-size-bytes=64mb`, in practice all position deletes end up in a single 64mb row group. Just lowering the size to 2mb in the example would make the uncached read 6.181/0.361=17 times faster. Row group size only affects caching performance at the 1mb setting. The table below shows cache vs. selective-read timing at different file_path counts and row group sizes:
| file_path count \ row group size | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 4 | 3.31x | 1.85x | 1.78x | 1.60x | 1.43x | 1.14x | 0.81x |
| 8 | 6.56x | 3.59x | 3.20x | 2.73x | 1.74x | 1.37x | 0.90x |
| 16 | 12.92x | 6.58x | 5.49x | 3.81x | 2.83x | 1.74x | 0.99x |
| 32 | 26.57x | 11.52x | 8.31x | 4.34x | 3.43x | 1.85x | 1.00x |
| 64 | 55.25x | 17.97x | 11.43x | 6.66x | 3.67x | 1.95x | 1.01x |
| 128 | 97.26x | 25.34x | 13.43x | 6.50x | 3.55x | 1.76x | 0.99x |
So, with a 2mb row group size and 64 referenced files (as in the example), caching pays off only past ~18 lookups.
---
This part is for the especially curious - for those who think anything over a second is slow for 64 megarows. If we add page pruning, filter before materialization, skip decoding file_path pages where min=max, and parallelize row group processing, caching becomes 2.3-7.5 times faster than on the PR branch. Selective read becomes 1.4-76.8 times faster, never crossing the 1-second mark. I still think there's room for improvement, though :)
<details>
<summary>Full benchmark results</summary>
| row group size | file_path count | read | read speedup | cache | cache speedup | cache vs read |
|---:|---:|---:|---:|---:|---:|---:|
| 4 | 4 | 0.528 | 6.64x | 0.836 | 7.45x | 1.58x |
| 4 | 8 | 0.464 | 4.20x | 0.836 | 7.45x | 1.80x |
| 4 | 16 | 0.441 | 2.57x | 0.836 | 7.45x | 1.90x |
| 4 | 32 | 0.351 | 2.14x | 0.836 | 7.45x | 2.38x |
| 4 | 64 | 0.343 | 1.59x | 0.836 | 7.45x | 2.44x |
| 4 | 128 | 0.335 | 1.39x | 0.836 | 7.45x | 2.49x |
| 8 | 4 | 0.601 | 6.48x | 0.862 | 7.23x | 1.43x |
| 8 | 8 | 0.454 | 5.03x | 0.862 | 7.23x | 1.90x |
| 8 | 16 | 0.294 | 5.57x | 0.862 | 7.23x | 2.93x |
| 8 | 32 | 0.218 | 6.58x | 0.862 | 7.23x | 3.95x |
| 8 | 64 | 0.184 | 5.08x | 0.862 | 7.23x | 4.68x |
| 8 | 128 | 0.168 | 5.71x | 0.862 | 7.23x | 5.13x |
| 16 | 4 | 0.764 | 5.71x | 1.033 | 6.03x | 1.35x |
| 16 | 8 | 0.430 | 8.32x | 1.033 | 6.03x | 2.40x |
| 16 | 16 | 0.247 | 8.91x | 1.033 | 6.03x | 4.18x |
| 16 | 32 | 0.159 | 11.43x | 1.033 | 6.03x | 6.49x |
| 16 | 64 | 0.119 | 14.25x | 1.033 | 6.03x | 8.68x |
| 16 | 128 | 0.096 | 18.29x | 1.033 | 6.03x | 10.76x |
| 32 | 4 | 0.724 | 7.54x | 1.610 | 3.87x | 2.22x |
| 32 | 8 | 0.400 | 11.35x | 1.610 | 3.87x | 4.03x |
| 32 | 16 | 0.216 | 16.60x | 1.610 | 3.87x | 7.45x |
| 32 | 32 | 0.137 | 24.55x | 1.610 | 3.87x | 11.75x |
| 32 | 64 | 0.092 | 34.80x | 1.610 | 3.87x | 17.50x |
| 32 | 128 | 0.075 | 47.11x | 1.610 | 3.87x | 21.47x |
| 64 | 4 | 0.739 | 10.43x | 2.691 | 2.32x | 3.64x |
| 64 | 8 | 0.420 | 16.45x | 2.691 | 2.32x | 6.41x |
| 64 | 16 | 0.230 | 27.50x | 2.691 | 2.32x | 11.70x |
| 64 | 32 | 0.143 | 43.40x | 2.691 | 2.32x | 18.82x |
| 64 | 64 | 0.098 | 63.07x | 2.691 | 2.32x | 27.46x |
| 64 | 128 | 0.082 | 76.78x | 2.691 | 2.32x | 32.82x |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456028754
For position deletes we had two layouts: partition-scoped and file-scoped (write.delete.granularity). File-scoped files only contain deletes for a single data file, so caching the whole file never helps a second data file. Our default for Spark is FILE scoped.
Partition-scoped files can contain deletes for many data files, but in practice they usually donāt grow that large. A write only sees the files modified in that job, and people rarely ran position-delete compaction (it landed well after v2 deletes). I donāt think I ever saw a position delete file get close to the 64 MB target. Equality deletes are the opposite: one file typically applies to every data file in the partition.
The other issue is executor locality. The cache is per JVM, so you only win if two tasks that share a position delete file land on the same executor. For equality deletes thatās almost guaranteed because metric based exclusion is gonna be rare unless you keep your files sorted on the equality delete columns. For position deletes on a normal scan itās mostly luck (unless we did more with scheduling / locality hints). Rewrite is a bit better within a file group, since those data files are read in one task, but groups in the same partition still go to different executors, so you still reload the delete file on each of them.
My big suggestion here is we just add a parameter (`cache-delete-files`) so that users like yourself can turn that caching back on manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456786429
I'm creating the PR as we speak. I will ping @nssalian (1.12.0 release manager) when it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456616278
> We would only ever know at scan time, we can't really rely on the table config and such.
Agreed.
One thing is still unclear to me, though: why do we cache position deletes for standard queries but skip caching for compaction? Or is the answer simply "don't ask, just upgrade to v3"? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SparkExecutorCache causes slowness of RewriteDataFilesSparkAction [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5457457604
Here is the PR: https://github.com/apache/iceberg/pull/17868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org