[GitHub] [hudi] codope commented on a diff in pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

["codope (via GitHub)" <gi...@apache.org>]

codope commented on code in PR #8041:
URL: https://github.com/apache/hudi/pull/8041#discussion_r1122575366


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java:
##########
@@ -40,27 +48,51 @@ public class Metrics {
   private static Metrics instance = null;
 
   private final MetricRegistry registry;
-  private MetricsReporter reporter;
+  private final List<MetricsReporter> reporters;
   private final String commonMetricPrefix;
 
   private Metrics(HoodieWriteConfig metricConfig) {
     registry = new MetricRegistry();
     commonMetricPrefix = metricConfig.getMetricReporterMetricsNamePrefix();
-    reporter = MetricsReporterFactory.createReporter(metricConfig, registry);
-    if (reporter == null) {
-      throw new RuntimeException("Cannot initialize Reporter.");
+    reporters = new ArrayList<>();
+    MetricsReporter defaultReporter = MetricsReporterFactory.createReporter(metricConfig, registry);
+    if (defaultReporter != null) {
+      reporters.add(defaultReporter);
     }
-    reporter.start();
+    reporters.addAll(addAdditionalMetricsExporters(metricConfig));
+    if (reporters.size() == 0) {
+      throw new RuntimeException("Cannot initialize Reporters.");
+    }
+    reporters.forEach(MetricsReporter::start);
 
     Runtime.getRuntime().addShutdownHook(new Thread(Metrics::shutdown));
   }
 
+  private List<MetricsReporter> addAdditionalMetricsExporters(HoodieWriteConfig metricConfig) {
+    List<MetricsReporter> reporterList = new ArrayList<>();
+    if (!StringUtils.isNullOrEmpty(metricConfig.getMetricReporterFileBasedConfigs())) {
+      List<String> propPathList = StringUtils.split(metricConfig.getMetricReporterFileBasedConfigs(), ",");
+      try (FileSystem fs = FSUtils.getFs(propPathList.get(0), new Configuration())) {
+        for (String propPath: propPathList) {
+          HoodieWriteConfig secondarySourceConfig = HoodieWriteConfig.newBuilder().fromInputStream(
+              fs.open(new Path(propPath))).withPath(metricConfig.getBasePath()).build();
+          reporterList.add(MetricsReporterFactory.createReporter(secondarySourceConfig, registry));
+        }
+      } catch (IOException e) {
+        LOG.error("Failed to add MetricsExporters", e);
+        throw new HoodieException("failed to MetricsExporters", e);

Review Comment:
   sounds good. even if we did throw an exception it should be handled in the caller.. continuous ingestion takes precedence over metrics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org