[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

["lokeshj1703 (via GitHub)" <gi...@apache.org>]

lokeshj1703 commented on code in PR #8041:
URL: https://github.com/apache/hudi/pull/8041#discussion_r1119632628


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java:
##########
@@ -40,27 +48,51 @@ public class Metrics {
   private static Metrics instance = null;
 
   private final MetricRegistry registry;
-  private MetricsReporter reporter;
+  private final List<MetricsReporter> reporters;
   private final String commonMetricPrefix;
 
   private Metrics(HoodieWriteConfig metricConfig) {
     registry = new MetricRegistry();
     commonMetricPrefix = metricConfig.getMetricReporterMetricsNamePrefix();
-    reporter = MetricsReporterFactory.createReporter(metricConfig, registry);
-    if (reporter == null) {
-      throw new RuntimeException("Cannot initialize Reporter.");
+    reporters = new ArrayList<>();
+    MetricsReporter defaultReporter = MetricsReporterFactory.createReporter(metricConfig, registry);
+    if (defaultReporter != null) {
+      reporters.add(defaultReporter);
     }
-    reporter.start();
+    reporters.addAll(addAdditionalMetricsExporters(metricConfig));
+    if (reporters.size() == 0) {
+      throw new RuntimeException("Cannot initialize Reporters.");
+    }
+    reporters.forEach(MetricsReporter::start);
 
     Runtime.getRuntime().addShutdownHook(new Thread(Metrics::shutdown));
   }
 
+  private List<MetricsReporter> addAdditionalMetricsExporters(HoodieWriteConfig metricConfig) {
+    List<MetricsReporter> reporterList = new ArrayList<>();
+    if (!StringUtils.isNullOrEmpty(metricConfig.getMetricReporterFileBasedConfigs())) {
+      List<String> propPathList = StringUtils.split(metricConfig.getMetricReporterFileBasedConfigs(), ",");
+      try (FileSystem fs = FSUtils.getFs(propPathList.get(0), new Configuration())) {
+        for (String propPath: propPathList) {
+          HoodieWriteConfig secondarySourceConfig = HoodieWriteConfig.newBuilder().fromInputStream(
+              fs.open(new Path(propPath))).withPath(metricConfig.getBasePath()).build();
+          reporterList.add(MetricsReporterFactory.createReporter(secondarySourceConfig, registry));
+        }
+      } catch (IOException e) {
+        LOG.error("Failed to add MetricsExporters", e);
+        throw new HoodieException("failed to MetricsExporters", e);

Review Comment:
   I guess we could ignore a connection issue but this error would occur while reading the file. This would be more of a setup issue and could lead to a silent failure. It usually would take some time to debug these issues. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org