You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "beyond1920 (via GitHub)" <gi...@apache.org> on 2023/08/31 15:02:29 UTC
[GitHub] [hudi] beyond1920 opened a new issue, #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
beyond1920 opened a new issue, #9591:
URL: https://github.com/apache/hudi/issues/9591
I found the resources could not be released in time in Spark compaction job.
In the following picture, only a few task are running, but all resources are not released even the tasks on them are all finished.
The root cause of this problem is write status RDD of compact is persist here.
<img width="1026" alt="image" src="https://github.com/apache/hudi/assets/1525333/ad2e28ed-53c4-4f2d-9ba3-fc957ec067a6">
I checked the code path of compaction. The write status RDD of compact only used once, and it would not be reused or triggered again(It is different with normal insert/upsert workflow, the write status of insert/upsert workflow would be used multiple times).
I set `hoodie.write.status.storage.level=NONE` in compaction job to avoid persist write status RDD. After disable persist the cost resource is only 30% of the original job which use default value (MEMORY_AND_DISK_SER). The befit is huge.
Could we remove the persist of write status RDD in compaction workflow, since it could decrease resource costs and the RDD is never used again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1713342671
@beyond1920 got it, in stage 0, if we have 1000 task run in 200 excutors, and now only has 1 long time task running in 1 excutor, other all success, but driver can not release these 199 excutors to Resource Manager because these excutor with the rdd cache, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1709695240
@beyond1920 you mean, the rdd cache is not unpersisted, and it is distributed on a large number of Executor, if use `resource dynamic allocation`, spark will not release these executor's resources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1710010726
@yihua
> If it's small, it should not cause much overhead and it's surprising that there is 70% of resource reduction after disabling persist.
The cached write status is not large. But if use persist, and default value of `hoodie.write.status.storage.level` is `MEMORY_AND_DISK_SER`, it would cause the executor could not released when the task is finished, because the cached in executor memory might be reused in next stage.
> Likely there's a bug causing the cached RDD not to be unpersisted once the job is done.
Not once the job is done, in fact we should say once the tasks on the executor is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1708825247
Hi @beyond1920 Thanks for raising this issue. Under the "Storage" tab in Spark UI, how large is the cached write status? If it's small, it should not cause much overhead and it's surprising that there is 70% of resource reduction after disabling persist. Likely there's a bug causing the cached RDD not to be unpersisted once the job is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1710014198
@KnightChess Sorry, I don't understand you quite well. Could you please explain in detail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1713778716
@KnightChess Exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1759690720
Is this problem solved? @beyond1920 @yihua I think delete the cache code can solve this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1701976717
cc @nsivabalan , guess the analysis is reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1760657182
@KnightChess Not yet. Waiting for confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1713086346
@yihua It's greate if the fix can be merged into release 0.14.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1710036286
@beyond1920 I mean, if executor which with the rdd cache, it will not be released by driver, look like hudi rdd cache not release in tableServiceClient, only release in writeClient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1713285421
Hi @yihua @KnightChess I think there are two bugs here.
1. The first bug is table service client does not properly release the cached RDDs. It would leads to the resources could not release in time after the job finished. It's the problem @yihua @KnightChess point out.
2. The second problem is it's not necessary to persist this write status RDD because the RDD would not reused again in the compaction job. Currently, the write status would persist anyway, it would leads to the cached RDD not to be unpersisted once all tasks on the executor is done. If there are slow tasks running in current stage, the other executors resources would not be released even if their tasks in this stage have already finished quickly. The 70% of resource reduction comes from this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-1712068499
@beyond1920 @KnightChess Yes, after looking into the code, it looks like the table service client does not properly release the cached RDDs. I'll provide a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #9591: [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time
URL: https://github.com/apache/hudi/issues/9591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] persist write status RDD in spark compaction job caused the resources could not be released in time [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #9591:
URL: https://github.com/apache/hudi/issues/9591#issuecomment-2348807843
https://github.com/apache/hudi/pull/11000
already fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org