You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/02/13 20:50:34 UTC
[GitHub] [hudi] jonvex opened a new pull request, #7934: metrics now offers multitable support
jonvex opened a new pull request, #7934:
URL: https://github.com/apache/hudi/pull/7934
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1428867003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1433612581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
}, {
"hash" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143)
* 2830ceca125d836eecfb129654d19a37aaf5fe3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #7934:
URL: https://github.com/apache/hudi/pull/7934#discussion_r1108902799
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java:
##########
@@ -52,75 +52,62 @@ private Metrics(HoodieWriteConfig metricConfig) {
}
reporter.start();
- Runtime.getRuntime().addShutdownHook(new Thread(Metrics::shutdown));
+ Runtime.getRuntime().addShutdownHook(new Thread(this::shutdown));
+ this.initialized = true;
}
- private void reportAndStopReporter() {
- try {
- registerHoodieCommonMetrics();
- reporter.report();
- LOG.info("Stopping the metrics reporter...");
- reporter.stop();
- } catch (Exception e) {
- LOG.warn("Error while closing reporter", e);
- }
+ private void registerHoodieCommonMetrics() {
+ registerGauges(Registry.getAllMetrics(true, true), Option.of(commonMetricPrefix));
}
- private void reportAndFlushMetrics() {
- try {
- LOG.info("Reporting and flushing all metrics");
- this.registerHoodieCommonMetrics();
- this.reporter.report();
- this.registry.getNames().forEach(this.registry::remove);
- } catch (Exception e) {
- LOG.error("Error while reporting and flushing metrics", e);
+ public static synchronized Metrics getInstance(HoodieWriteConfig metricConfig) {
+ String basePath = metricConfig.getBasePath();
+ if (METRICS_INSTANCE_PER_BASEPATH.containsKey(basePath)) {
+ return METRICS_INSTANCE_PER_BASEPATH.get(basePath);
}
- }
- private void registerHoodieCommonMetrics() {
- registerGauges(Registry.getAllMetrics(true, true), Option.of(commonMetricPrefix));
+ Metrics metrics = new Metrics(metricConfig);
+ METRICS_INSTANCE_PER_BASEPATH.put(basePath, metrics);
+ return metrics;
}
- public static Metrics getInstance() {
- assert initialized;
- return instance;
+ public static synchronized void shutdownAllMetrics() {
+ METRICS_INSTANCE_PER_BASEPATH.values().forEach(Metrics::shutdown);
}
- public static synchronized void init(HoodieWriteConfig metricConfig) {
- if (initialized) {
- return;
- }
+ public synchronized void shutdown() {
try {
- instance = new Metrics(metricConfig);
+ registerHoodieCommonMetrics();
+ reporter.report();
+ LOG.info("Stopping the metrics reporter...");
+ if (reporter != null) {
Review Comment:
Yeah that's not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #7934:
URL: https://github.com/apache/hudi/pull/7934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1433620081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
}, {
"hash" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254",
"triggerID" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143)
* 2830ceca125d836eecfb129654d19a37aaf5fe3b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1439246889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
}, {
"hash" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254",
"triggerID" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"triggerType" : "PUSH"
}, {
"hash" : "0afb2b3c974a716c33c9d74f816eb027ccb056a2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15323",
"triggerID" : "0afb2b3c974a716c33c9d74f816eb027ccb056a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0afb2b3c974a716c33c9d74f816eb027ccb056a2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15323)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1428729287
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1428721251
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1439094201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
}, {
"hash" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254",
"triggerID" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"triggerType" : "PUSH"
}, {
"hash" : "0afb2b3c974a716c33c9d74f816eb027ccb056a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0afb2b3c974a716c33c9d74f816eb027ccb056a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2830ceca125d836eecfb129654d19a37aaf5fe3b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254)
* 0afb2b3c974a716c33c9d74f816eb027ccb056a2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7934:
URL: https://github.com/apache/hudi/pull/7934#discussion_r1108725732
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java:
##########
@@ -52,75 +52,62 @@ private Metrics(HoodieWriteConfig metricConfig) {
}
reporter.start();
- Runtime.getRuntime().addShutdownHook(new Thread(Metrics::shutdown));
+ Runtime.getRuntime().addShutdownHook(new Thread(this::shutdown));
+ this.initialized = true;
}
- private void reportAndStopReporter() {
- try {
- registerHoodieCommonMetrics();
- reporter.report();
- LOG.info("Stopping the metrics reporter...");
- reporter.stop();
- } catch (Exception e) {
- LOG.warn("Error while closing reporter", e);
- }
+ private void registerHoodieCommonMetrics() {
+ registerGauges(Registry.getAllMetrics(true, true), Option.of(commonMetricPrefix));
}
- private void reportAndFlushMetrics() {
- try {
- LOG.info("Reporting and flushing all metrics");
- this.registerHoodieCommonMetrics();
- this.reporter.report();
- this.registry.getNames().forEach(this.registry::remove);
- } catch (Exception e) {
- LOG.error("Error while reporting and flushing metrics", e);
+ public static synchronized Metrics getInstance(HoodieWriteConfig metricConfig) {
+ String basePath = metricConfig.getBasePath();
+ if (METRICS_INSTANCE_PER_BASEPATH.containsKey(basePath)) {
+ return METRICS_INSTANCE_PER_BASEPATH.get(basePath);
}
- }
- private void registerHoodieCommonMetrics() {
- registerGauges(Registry.getAllMetrics(true, true), Option.of(commonMetricPrefix));
+ Metrics metrics = new Metrics(metricConfig);
+ METRICS_INSTANCE_PER_BASEPATH.put(basePath, metrics);
+ return metrics;
}
- public static Metrics getInstance() {
- assert initialized;
- return instance;
+ public static synchronized void shutdownAllMetrics() {
+ METRICS_INSTANCE_PER_BASEPATH.values().forEach(Metrics::shutdown);
}
- public static synchronized void init(HoodieWriteConfig metricConfig) {
- if (initialized) {
- return;
- }
+ public synchronized void shutdown() {
try {
- instance = new Metrics(metricConfig);
+ registerHoodieCommonMetrics();
+ reporter.report();
+ LOG.info("Stopping the metrics reporter...");
+ if (reporter != null) {
Review Comment:
how can reporter could be null. Just 1 line above we are calling reporter.report() and this is within synchronized method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1433348392
why is this a draft ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7934: [HUDI-5777] Support Metrics for Multiple Tables Simultaneously
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7934:
URL: https://github.com/apache/hudi/pull/7934#issuecomment-1434130781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15143",
"triggerID" : "63ab1c1d6bf7adecea5a59f97c6d0e52648cf0f1",
"triggerType" : "PUSH"
}, {
"hash" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254",
"triggerID" : "2830ceca125d836eecfb129654d19a37aaf5fe3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2830ceca125d836eecfb129654d19a37aaf5fe3b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org