You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sandeep-heap (via GitHub)" <gi...@apache.org> on 2023/09/19 06:37:24 UTC
[GitHub] [hudi] sandeep-heap opened a new issue, #9747: [SUPPORT]
sandeep-heap opened a new issue, #9747:
URL: https://github.com/apache/hudi/issues/9747
**Describe the problem you faced**
Async Cleaner OOM / slowdown after creating a large Savepoint
**To Reproduce**
Steps to reproduce the behavior:
1. Create a large savepoint e.g. 2GB,
2. Run an async clean by submitting a spark job with class `org.apache.hudi.utilities.HoodieCleaner` and args:
```
--emr_executor_cores=2
--emr_executor_instances=200
--emr_executor_memory=16G
--emr_driver_memory=40G
--target-base-path=s3://...
--hoodie.metadata.enable=false
--hoodie.cleaner.policy=KEEP_LATEST_COMMITS
--hoodie.cleaner.commits.retained=500
--hoodie.keep.min.commits=510
--hoodie.keep.max.commits=520
--hoodie.cleaner.policy.failed.writes=LAZY
--hoodie.cleaner.parallelism=1000
--hoodie.clean.allow.multiple=false
--hoodie.embed.timeline.server=false
--hoodie.archive.async=false
--hoodie.archive.automatic=false
```
**Expected behavior**
Hudi should successfully clean older commits. Before the savepoint, each run used to take ~2-3h.
**Environment Description**
* Hudi version : 0.13.0
* Spark version : Spark 3.3.0
* Hive version : N/A
* Hadoop version : Hadoop 3.2.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
After taking a memory dump using java option `HeapDumpOnOutOfMemoryError` we note the following:
- Biggest class is `byte[]` which take up >80% of the snapshot,
- Instances vary in size from 268MB to 67MB e.g. 94 instances =~ 13GB
- Referenced by `java.io.ByteArrayOutputStream, GC Root: Java Frame: org.apache.hudi.common.util.FileIOUtils.copy(FileIOUtils.java:101)`
- `protected byte buf[];` var contains avro record of type `HoodieSavepointMetadata`
Increasing spark memory to ~300GB did not alleviate the OOMs. Reducing `hoodie.cleaner.parallelism` to `40` finally brought memory consumption down but unfortunately degraded performance. Step `Generating list of file slices to be cleaned` is taking days to run the following code:
```
// org.apache.hudi.table.action.clean.CleanPlanner.getFilesToCleanKeepingLatestCommits(CleanPlanner.java:332)
if (aFile.isPresent() && savepointedFiles.contains(aFile.get().getFileName())) {
// do not clean up a savepoint data file
continue;
}
```
**Stacktrace**
Stack trace before OOM:
```
Thread 'Executor task launch worker for task 532.1 in stage 0.0 (TID 1781)' with ID = 142
java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:232)
sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:190)
sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1404)
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1372)
sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:966)
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:197)
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125)
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
com.amazon.ws.emr.hadoop.fs.s3.InputStreamWithInfo.read(InputStreamWithInfo.java:231)
com.amazon.ws.emr.hadoop.fs.s3.S3FSInputStream.read(S3FSInputStream.java:262)
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
java.io.BufferedInputStream.read(BufferedInputStream.java:345)
java.io.DataInputStream.read(DataInputStream.java:149)
java.io.DataInputStream.read(DataInputStream.java:100)
org.apache.hudi.common.util.FileIOUtils.copy(FileIOUtils.java:101)
org.apache.hudi.common.util.FileIOUtils.readAsByteArray(FileIOUtils.java:137)
org.apache.hudi.common.util.FileIOUtils.readAsByteArray(FileIOUtils.java:132)
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.readDataFromPath(HoodieActiveTimeline.java:822)
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.getInstantDetails(HoodieActiveTimeline.java:310)
org.apache.hudi.table.action.clean.CleanPlanner.getSavepointedDataFiles(CleanPlanner.java:118)
org.apache.hudi.table.action.clean.CleanPlanner$$Lambda$2551.apply(Native method)
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
org.apache.hudi.table.action.clean.CleanPlanner.getFilesToCleanKeepingLatestCommits(CleanPlanner.java:305)
org.apache.hudi.table.action.clean.CleanPlanner.getFilesToCleanKeepingLatestCommits(CleanPlanner.java:278)
org.apache.hudi.table.action.clean.CleanPlanner.getDeletePaths(CleanPlanner.java:448)
org.apache.hudi.table.action.clean.CleanPlanActionExecutor.lambda$requestClean$e73901b4$1(CleanPlanActionExecutor.java:121)
org.apache.hudi.table.action.clean.CleanPlanActionExecutor$$Lambda$2309.apply(Native method)
org.apache.hudi.client.common.HoodieSparkEngineContext$$Lambda$2311.call(Native method)
...
```
Stack trace during slow step `Generating list of file slices to be cleaned`:
```
org.apache.hudi.table.action.clean.CleanPlanner.getFilesToCleanKeepingLatestCommits(CleanPlanner.java:332)
org.apache.hudi.table.action.clean.CleanPlanner.getFilesToCleanKeepingLatestCommits(CleanPlanner.java:278)
org.apache.hudi.table.action.clean.CleanPlanner.getDeletePaths(CleanPlanner.java:448)
org.apache.hudi.table.action.clean.CleanPlanActionExecutor.lambda$requestClean$e73901b4$1(CleanPlanActionExecutor.java:121)
org.apache.hudi.table.action.clean.CleanPlanActionExecutor$$Lambda$1963/1842056606.apply(Unknown Source)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9747: [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1728032002
@sandeep-heap Did you tried optimising the spark memory configs? Also can you confirm if executor-memory is being set properly. (may be in spark UI , as 16gb also should work)
Which cluster setup are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ehurheap commented on issue #9747: [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint
Posted by "ehurheap (via GitHub)" <gi...@apache.org>.
ehurheap commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1739881993
@ad1happy2go, as I mentioned in office hours this morning, our MOR table has just over 100 million files and contains just over 1 PB of data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] conorzhao commented on issue #9747: [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint
Posted by "conorzhao (via GitHub)" <gi...@apache.org>.
conorzhao commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1736591814
I also encountered the issue of OOM when Flink JM runs for a period of time. My usage scenario is to synchronize the entire library of Flink cdc MySQL to Hudi, with over 100 tables in one library
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1778662842
@ehurheap Had a discussion, It will involve a bigger change in save point design to make it as a distributed dataset and then read it using spark.
Created JIRA - https://issues.apache.org/jira/browse/HUDI-6981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sandeep-heap commented on issue #9747: [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint
Posted by "sandeep-heap (via GitHub)" <gi...@apache.org>.
sandeep-heap commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1728745545
Hi @ad1happy2go we've tried sevaral memory configurations e.g. from 8GB all the way up to 300GB. Confirmed that memory is being used via Ganglia, Spark UI and memory dumps. No of executors doesn't seem to make a difference -- just the `hoodie.cleaner.parallelism` reduces the memory footprint.
We are using AWS EMR on EC2 to run the job.
From what I can glean in the code, it looks like each cleaner thread loads all savepoints into memory: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java#L248-L251
Given our large Savepoint size (~2GB on S3), wouldn't this cause the behaviour I've described? It also looks like the savepoint file is downloaded again for each parition path processed. Given the high no of partitions in our setup, this also seems to have a negative effect on performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9747: [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1738825740
@sandeep-heap I guess you are right, such a large save point can cause OOM issues. How your savepoint size is so large as it doesn't contain the data but only file names. Do you have those number of files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1771164682
@sandeep-heap @ehurheap Having internal discussions on this on how can we handle such large save points. Will update you soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint [hudi]
Posted by "ehurheap (via GitHub)" <gi...@apache.org>.
ehurheap commented on issue #9747:
URL: https://github.com/apache/hudi/issues/9747#issuecomment-1779720173
Interesting. Thanks for the update @ad1happy2go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #9747: [SUPPORT] Async Cleaner OOM / slowdown after creating a large Savepoint
URL: https://github.com/apache/hudi/issues/9747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org