You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stream2000 (via GitHub)" <gi...@apache.org> on 2023/08/24 08:02:20 UTC
[GitHub] [hudi] stream2000 opened a new pull request, #9515: [HUDI-2141] Support flink compaction metrics
stream2000 opened a new pull request, #9515:
URL: https://github.com/apache/hudi/pull/9515
### Change Logs
Support flink compaction metrics
sub-task for HUDI-2141
### Impact
Support flink compaction metrics
### Risk level (write none, low medium or high below)
low
### Documentation Update
Will update documents in another pr
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1691360896
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435)
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1691373305
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435)
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1312437663
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
Yes of course. Will delete them later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1706632630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636)
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1309766178
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
Yeah, reasonable. I'm just not very convinced we need so many metrics here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1311016199
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
I think we can make use of these commit metrics in the near future once [flink support coordinator metrics](https://cwiki.apache.org/confluence/display/FLINK/FLIP-274%3A+Introduce+metric+group+for+OperatorCoordinator). SO we can keep them now sothat we won't do large modification if we can gather metrics from `StreamwriteCoordinator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1691348730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1692716707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1706934980
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1708251534
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1309766178
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
Yeah, reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1693080271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451)
* 33ea8bad45355a5cfb69955f372f0e3a87540aae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1702097038
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33ea8bad45355a5cfb69955f372f0e3a87540aae Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466)
* a11cc23103021a2916d2759bead59b61a80e50f7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704590706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1693016920
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451)
* 33ea8bad45355a5cfb69955f372f0e3a87540aae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1703844579
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a11cc23103021a2916d2759bead59b61a80e50f7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614)
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704200168
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1703890500
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1692712087
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 closed pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 closed pull request #9515: [HUDI-2141] Support flink compaction metrics
URL: https://github.com/apache/hudi/pull/9515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1691277077
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1691601687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704509324
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1702091623
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33ea8bad45355a5cfb69955f372f0e3a87540aae Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466)
* a11cc23103021a2916d2759bead59b61a80e50f7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1707796296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669",
"triggerID" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654)
* 89ffa9c15288860c420fe2d380131d9ee3d61c21 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1312435023
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
Can we drop these write metrics first until we introduce the coordinator metrics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1706712590
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636)
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1707745041
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654)
* 89ffa9c15288860c420fe2d380131d9ee3d61c21 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1307089689
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
> Can this class be reused by Flink?
Flink has its own metrics system and it seems hard to reuse hudi-spark's metrics system. For example, flink metrics core manages the `MetricsReporter` itself and we can config metrics reporters through flink config. However for hudi spark, we manager the reporter in hudi instead of Spark. We do can reuse the metrics defined in `hudi-client-common` for flink, however in this way we will bypass Flink's metrics system which may make hudi-flink metrics different from other flink metrics.
I think in hudi-flink, we'd better use flink's metrics system so that those who already have a metrics for flink can easily integrate hudi-flink metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1306838878
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
> Do we need all these metrics?
These metrics are common commit metrics and not all are useful in compaction. However, if we can gather metrics after the write coordinator commit in the future, these metrics will be useful. We can keep them for future extensions or add them back when needed.
> How Spark handles these metrics?
Spark maintain a large metrics class and will gather these metrics for all write (clustering, compaction, common writes..). We can align with spark and keep these metrics. If we add some metrics to spark commit metrics, we can migrate them to flink easily and don't need do a large diff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1306832966
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
Do we need all these metrics? How Spark handles these metrics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1317007650
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/CompactFunctionWrapper.java:
##########
@@ -102,11 +108,10 @@ public void openFunction() throws Exception {
public void compact(long checkpointID) throws Exception {
// collect the CompactEvents.
- CollectorOutput<CompactionPlanEvent> output = new CollectorOutput<>();
- compactionPlanOperator.setOutput(output);
+ compactionPlanOperator.setOutput(planEventOutput);
Review Comment:
to get RuntimeContext in `CompactionPlanOperator` in test. Current implementation will return null when we call `getRuntimeContext`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1316995233
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/CompactFunctionWrapper.java:
##########
@@ -102,11 +108,10 @@ public void openFunction() throws Exception {
public void compact(long checkpointID) throws Exception {
// collect the CompactEvents.
- CollectorOutput<CompactionPlanEvent> output = new CollectorOutput<>();
- compactionPlanOperator.setOutput(output);
+ compactionPlanOperator.setOutput(planEventOutput);
Review Comment:
why this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704131689
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9515:
URL: https://github.com/apache/hudi/pull/9515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1693171244
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33ea8bad45355a5cfb69955f372f0e3a87540aae Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1306945906
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
> a large metrics class
Can this class be reused by Flink?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1692928905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1703610601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a11cc23103021a2916d2759bead59b61a80e50f7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1703549261
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1703682906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a11cc23103021a2916d2759bead59b61a80e50f7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1703846504
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a11cc23103021a2916d2759bead59b61a80e50f7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614)
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1708239477
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669",
"triggerID" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89ffa9c15288860c420fe2d380131d9ee3d61c21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1708252802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669",
"triggerID" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19674",
"triggerID" : "1708251534",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 89ffa9c15288860c420fe2d380131d9ee3d61c21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19674)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1691290492
@danny0405 Hey Danny, could you help review this pr? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9515:
URL: https://github.com/apache/hudi/pull/9515#discussion_r1306838878
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
> Do we need all these metrics?
These metrics are common commit metrics and not all are useful in compaction. However, if we can gather metrics after the write coordinator commit in the future, these metrics will be useful. We can keep them for future extensions or add them back when needed.
> How Spark handles these metrics?
Spark maintain a large metrics class and will gather these metrics for all write (clustering, compaction, common writes..). We can align with spark and keep these metrics. If we add some metrics to spark commit metrics, we can migrate them to flink easily and don't need do a large diff
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkWriteMetrics.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.flink.metrics.MetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+/**
+ * Common flink write commit metadata metrics
+ */
+public class FlinkWriteMetrics extends HoodieFlinkMetrics {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkWriteMetrics.class);
+
+ protected final String actionType;
+
+ private long totalPartitionsWritten;
+ private long totalFilesInsert;
+ private long totalFilesUpdate;
+ private long totalRecordsWritten;
+ private long totalUpdateRecordsWritten;
+ private long totalInsertRecordsWritten;
+ private long totalBytesWritten;
+ private long totalScanTime;
+ private long totalCreateTime;
+ private long totalUpsertTime;
+ private long totalCompactedRecordsUpdated;
+ private long totalLogFilesCompacted;
+ private long totalLogFilesSize;
+ private long commitLatencyInMs;
+ private long commitFreshnessInMs;
+ private long commitEpochTimeInMs;
+ private long durationInMs;
+
+ public FlinkWriteMetrics(MetricGroup metricGroup, String actionType) {
+ super(metricGroup);
+ this.actionType = actionType;
+ }
+
+ @Override
+ public void registerMetrics() {
+ // register commit gauge
+ metricGroup.gauge(getMetricsName(actionType, "totalPartitionsWritten"), () -> totalPartitionsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesInsert"), () -> totalFilesInsert);
+ metricGroup.gauge(getMetricsName(actionType, "totalFilesUpdate"), () -> totalFilesUpdate);
+ metricGroup.gauge(getMetricsName(actionType, "totalRecordsWritten"), () -> totalRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpdateRecordsWritten"), () -> totalUpdateRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalInsertRecordsWritten"), () -> totalInsertRecordsWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalBytesWritten"), () -> totalBytesWritten);
+ metricGroup.gauge(getMetricsName(actionType, "totalScanTime"), () -> totalScanTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCreateTime"), () -> totalCreateTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalUpsertTime"), () -> totalUpsertTime);
+ metricGroup.gauge(getMetricsName(actionType, "totalCompactedRecordsUpdated"), () -> totalCompactedRecordsUpdated);
Review Comment:
> Do we need all these metrics?
These metrics are common commit metrics and not all are useful in compaction. However, if we can gather metrics after the write coordinator commit in the future, these metrics will be useful. We can keep them for future extensions or add them back when needed.
> How Spark handles these metrics?
Spark maintain a large metrics class and will gather these metrics for all write (clustering, compaction, common writes..). We can align with spark and keep these metrics. If we add some metrics to spark commit metrics, we can migrate them to flink easily and don't need do a large diff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1702051289
Tested locally and only these four metrics are useless. Remove them until we support coordinator metrics. @danny0405 What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1702127608
> Tested locally and only these four metrics are useless. Remove them until we support coordinator metrics. @danny0405 What do you think?
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1702499151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a11cc23103021a2916d2759bead59b61a80e50f7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1708693130
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19654",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669",
"triggerID" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"triggerType" : "PUSH"
}, {
"hash" : "89ffa9c15288860c420fe2d380131d9ee3d61c21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19674",
"triggerID" : "1708251534",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 89ffa9c15288860c420fe2d380131d9ee3d61c21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19669) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19674)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704516284
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636",
"triggerID" : "1704509324",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19636)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704599758
@danny0405 Ci is green now, would you like to take a final pass~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9515:
URL: https://github.com/apache/hudi/pull/9515#issuecomment-1704331108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19435",
"triggerID" : "f7b7116d5551f5e25d14f2ccf1b98baa6eb8b8b8",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19436",
"triggerID" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"triggerType" : "PUSH"
}, {
"hash" : "4ccfb5f43f82a7c57de83794ebaa9c5a051c679a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19451",
"triggerID" : "1692712087",
"triggerType" : "MANUAL"
}, {
"hash" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19466",
"triggerID" : "33ea8bad45355a5cfb69955f372f0e3a87540aae",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19597",
"triggerID" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"triggerType" : "PUSH"
}, {
"hash" : "a11cc23103021a2916d2759bead59b61a80e50f7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19614",
"triggerID" : "1703549261",
"triggerType" : "MANUAL"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622",
"triggerID" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"triggerType" : "PUSH"
}, {
"hash" : "4161e480bdc7d89254bf38c2f7df00fbad2b214e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634",
"triggerID" : "1704131689",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4161e480bdc7d89254bf38c2f7df00fbad2b214e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19622) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19634)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org