You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/02/06 01:02:38 UTC
[PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
nsivabalan opened a new pull request, #12789:
URL: https://github.com/apache/hudi/pull/12789
### Change Logs
Fixing source read and index tagging duration.
### Impact
Fixing source read and index tagging duration.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2657006915
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
}, {
"hash" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579",
"triggerID" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"triggerType" : "PUSH"
}, {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579)
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2660236792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
}, {
"hash" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617)
* 84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1956648405
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/commit/BaseFlinkCommitActionExecutor.java:
##########
@@ -90,12 +91,22 @@ public BaseFlinkCommitActionExecutor(HoodieEngineContext context,
@Override
public HoodieWriteMetadata<List<WriteStatus>> execute(List<HoodieRecord<T>> inputRecords) {
+ return execute(inputRecords, Option.empty());
+ }
+
+ @Override
+ public HoodieWriteMetadata<List<WriteStatus>> execute(List<HoodieRecord<T>> inputRecords, Option<HoodieTimer> sourceReadAndIndexTimer) {
HoodieWriteMetadata<List<WriteStatus>> result = new HoodieWriteMetadata<>();
List<WriteStatus> writeStatuses = new LinkedList<>();
final HoodieRecord<?> record = inputRecords.get(0);
final String partitionPath = record.getPartitionPath();
final String fileId = record.getCurrentLocation().getFileId();
+ Long sourceReadAndIndexDurationMs = null;
+ if (sourceReadAndIndexTimer.isPresent()) {
+ sourceReadAndIndexDurationMs = sourceReadAndIndexTimer.get().endTimer();
+ LOG.info("Source read and index timer " + sourceReadAndIndexDurationMs);
Review Comment:
nit {}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2646796821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70e5ceef3f4eea079b6898a02f7c9c21ac6b5262 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407)
* 671cb50ad9f251d2286dca8d8b4b0978b5696046 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2646798553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70e5ceef3f4eea079b6898a02f7c9c21ac6b5262 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407)
* 671cb50ad9f251d2286dca8d8b4b0978b5696046 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1948329538
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
Review Comment:
I moved it to the beginning of
BaseWriteHelper.write
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2638375715
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70e5ceef3f4eea079b6898a02f7c9c21ac6b5262 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2655122849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
}, {
"hash" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579",
"triggerID" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2638373921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70e5ceef3f4eea079b6898a02f7c9c21ac6b5262 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2654991242
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
}, {
"hash" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 671cb50ad9f251d2286dca8d8b4b0978b5696046 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507)
* 3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2667458931
@danny0405 : reverted changes in flink and java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2661850348
> So, why revert from flink?
@nsivabalan Because Flink has fixed pipeline and the timer metric does not make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2638397223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70e5ceef3f4eea079b6898a02f7c9c21ac6b5262 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1955490951
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
Review Comment:
Flink does not has the issue like lazy computation of Spark, so the timer does not make much sense, I think this is also true for Java, maybe we revert the changes for Flink and JAVA and just Spark as the minimum change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2646929165
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 671cb50ad9f251d2286dca8d8b4b0978b5696046 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2660175335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2660275729
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
}, {
"hash" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644",
"triggerID" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617)
* 84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2657299000
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
}, {
"hash" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579",
"triggerID" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"triggerType" : "PUSH"
}, {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2667547829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
}, {
"hash" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644",
"triggerID" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"triggerType" : "PUSH"
}, {
"hash" : "55e5dbc42d81752f6979b2d28d325f177ac14c3b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3694",
"triggerID" : "55e5dbc42d81752f6979b2d28d325f177ac14c3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 55e5dbc42d81752f6979b2d28d325f177ac14c3b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3694)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #12789:
URL: https://github.com/apache/hudi/pull/12789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1948344371
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
Review Comment:
Looks much better now, just a minor comment: in base executor the `execute(I inputRecords, Option<HoodieTimer> preWriteTimer)` invokes `execute(I inputRecords)` but in it's Spark impl it's the other way around, this is kind of tricky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2657012177
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
}, {
"hash" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579",
"triggerID" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"triggerType" : "PUSH"
}, {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579)
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1956651767
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
new WorkloadProfile(buildProfile(inputRecordsWithClusteringUpdate), operationType, table.getIndex().canIndexLogFiles());
LOG.debug("Input workload profile :" + workloadProfile);
- long sourceReadAndIndexDurationMs = sourceReadAndIndexTimer.endTimer();
- LOG.info("Source read and index timer " + sourceReadAndIndexDurationMs);
-
+ Long sourceReadAndIndexDurationMs = null;
+ if (sourceReadAndIndexTimer.isPresent()) {
+ sourceReadAndIndexDurationMs = sourceReadAndIndexTimer.get().endTimer();
+ LOG.info("Source read and index timer " + sourceReadAndIndexDurationMs);
Review Comment:
nit {}
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
new WorkloadProfile(buildProfile(inputRecordsWithClusteringUpdate), operationType, table.getIndex().canIndexLogFiles());
LOG.debug("Input workload profile :" + workloadProfile);
Review Comment:
nit {}
##########
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java:
##########
@@ -86,12 +87,21 @@ public BaseJavaCommitActionExecutor(HoodieEngineContext context,
@Override
public HoodieWriteMetadata<List<WriteStatus>> execute(List<HoodieRecord<T>> inputRecords) {
+ return execute(inputRecords, Option.empty());
+ }
+
+ @Override
+ public HoodieWriteMetadata<List<WriteStatus>> execute(List<HoodieRecord<T>> inputRecords, Option<HoodieTimer> sourceReadAndIndexTimer) {
HoodieWriteMetadata<List<WriteStatus>> result = new HoodieWriteMetadata<>();
WorkloadProfile workloadProfile =
new WorkloadProfile(buildProfile(inputRecords), table.getIndex().canIndexLogFiles());
LOG.info("Input workload profile :" + workloadProfile);
-
+ Long sourceReadAndIndexDurationMs = null;
+ if (sourceReadAndIndexTimer.isPresent()) {
+ sourceReadAndIndexDurationMs = sourceReadAndIndexTimer.get().endTimer();
+ LOG.info("Source read and index timer " + sourceReadAndIndexDurationMs);
Review Comment:
nit {}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2660421290
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
}, {
"hash" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644",
"triggerID" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1954756763
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
Review Comment:
addressed for all engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2654994123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3407",
"triggerID" : "70e5ceef3f4eea079b6898a02f7c9c21ac6b5262",
"triggerType" : "PUSH"
}, {
"hash" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507",
"triggerID" : "671cb50ad9f251d2286dca8d8b4b0978b5696046",
"triggerType" : "PUSH"
}, {
"hash" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579",
"triggerID" : "3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 671cb50ad9f251d2286dca8d8b4b0978b5696046 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3507)
* 3c2b6e4b8d3ad9b9690e956b8f561a1964f5077a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3579)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2660011139
<img width="1207" alt="image" src="https://github.com/user-attachments/assets/1753f65d-305e-4949-8325-3097dd18dc66" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2660173235
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12789:
URL: https://github.com/apache/hudi/pull/12789#discussion_r1944010807
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -161,13 +166,14 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> execute(HoodieData<HoodieRec
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate = clusteringHandleUpdate(inputRecords);
context.setJobStatus(this.getClass().getSimpleName(), "Building workload profile:" + config.getTableName());
- HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start(); // time taken from dedup -> tag location -> building workload profile
WorkloadProfile workloadProfile =
Review Comment:
The new changes just initializes the timer on the fly and passed in, why not just move the timer at line 164 to the begining of the method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2667460206
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
}, {
"hash" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644",
"triggerID" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"triggerType" : "PUSH"
}, {
"hash" : "55e5dbc42d81752f6979b2d28d325f177ac14c3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3694",
"triggerID" : "55e5dbc42d81752f6979b2d28d325f177ac14c3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644)
* 55e5dbc42d81752f6979b2d28d325f177ac14c3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3694)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2667458497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3617",
"triggerID" : "48ee1ec0e7eb9138fe44e1bbc6e564308a3756bd",
"triggerType" : "PUSH"
}, {
"hash" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644",
"triggerID" : "84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217",
"triggerType" : "PUSH"
}, {
"hash" : "55e5dbc42d81752f6979b2d28d325f177ac14c3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "55e5dbc42d81752f6979b2d28d325f177ac14c3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84c2c048ae1ed7a2ed6b2c15210b0c9d49ae5217 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3644)
* 55e5dbc42d81752f6979b2d28d325f177ac14c3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7624] Fixing source read and index tagging duration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12789:
URL: https://github.com/apache/hudi/pull/12789#issuecomment-2661477902
@danny0405 : why? even though the fix is mainly for spark(due to lazy eval), still the contract of meaning of metric holds good for any engine. ie. total time taken to read from source and indexing. So, why revert from flink?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org