You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 11:57:37 UTC
[I] Fix HUDI-9130 which causes perf regression [hudi]
hudi-bot opened a new issue, #16917:
URL: https://github.com/apache/hudi/issues/16917
We realized that after we landed [https://github.com/apache/hudi/pull/13044] , we found 19% perf regression w/ spark-sql writes (bulk insert). We saw additional map and WholeStageCodeGen stages in sparkUI.
!image (14).png!
1_0_1
!image (16).png!
latest master
!image (15).png!
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9310
- Type: Improvement
- Fix version(s):
- 1.1.0
- Attachment(s):
- 13/Apr/25 22:54;shivnarayan;image (14).png;https://issues.apache.org/jira/secure/attachment/13075976/image+%2814%29.png
- 13/Apr/25 22:54;shivnarayan;image (15).png;https://issues.apache.org/jira/secure/attachment/13075977/image+%2815%29.png
- 13/Apr/25 22:54;shivnarayan;image (16).png;https://issues.apache.org/jira/secure/attachment/13075978/image+%2816%29.png
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix HUDI-9130 which causes perf regression [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16917:
URL: https://github.com/apache/hudi/issues/16917#issuecomment-4800682895
This appears to have been fixed by [PR #13139](https://github.com/apache/hudi/pull/13139) ([HUDI-9309] Fix perf regression from supporting display dag of insert/update statment in spark ui), commit `b91183d764b0`. That change replaced the `sparkAdapter.createSparkRowSerDe(...)` + `query.execute().map(sparkRowSerDe.deserializeRow)` round-trip introduced by #13044 with `sparkSession.internalCreateDataFrame(query.execute(), schema)` in both `InsertIntoHoodieTableCommand` and `UpdateHoodieTableCommand`, eliminating the extra map/WholeStageCodegen stages that caused the 19% regression. Current master (spark3-common and spark4-common variants) uses `internalCreateDataFrame` at line 119 of `InsertIntoHoodieTableCommand.scala`. Should this issue be closed? (Note: the JIRA key referenced in the issue body is HUDI-9310, but the fix landed under HUDI-9309 — likely the same ticket.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix HUDI-9130 which causes perf regression [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16917: Fix HUDI-9130 which causes perf regression
URL: https://github.com/apache/hudi/issues/16917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org