You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:00:37 UTC
[I] Revisiting doWriteOperation for preparing test data using java metadata table writer [hudi]
hudi-bot opened a new issue, #15516:
URL: https://github.com/apache/hudi/issues/15516
[This method|https://github.com/apache/hudi/blob/master/hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java#L910] in HoodieTestTable is used to create commits and some test methods will create over 10 commits. Each call is taking 3-4 seconds locally for me so if we could cut this down to 1-2 seconds we would see a big testing performance improvement.
public HoodieCommitMetadata doWriteOperation(String commitTime, WriteOperationType operationType,
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5093
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5197
---
## Comments
09/Nov/22 13:15;xushiyan;The time-consuming part mainly comes from `org.apache.hudi.common.testutils.HoodieMetadataTestTable#doWriteOperation` which invokes spark metadata writer to update metadata table upon a new commit.
This is a necessary process. I don't think there is much room to optimize here. If we implement a java metadata writer, it may be faster but lose coverage around spark metadata writer, which is the major use case.
WDYT? [~guoyihua][~shivnarayan];;;
---
11/Nov/22 10:35;xushiyan;as discussed, we should tackle this with java writer implementation for data prep. will push this for future improvements.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Revisiting doWriteOperation for preparing test data using java metadata table writer [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #15516:
URL: https://github.com/apache/hudi/issues/15516#issuecomment-5068106187
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: This is a test-infrastructure improvement (HUDI-5093) to speed up test-data preparation in HoodieTestTable.doWriteOperation, which currently invokes the Spark metadata-table writer on every commit (~3-4s each). As noted on the original JIRA, this was deferred to a future Java-metadata-writer implementation for test-data prep, and there is no linked fix PR. It remains an open, incomplete improvement rather than a bug.
Keeping open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org