[jira] [Updated] (HUDI-2404) JMX Reporter broken using spark package

["sivabalan narayanan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan updated HUDI-2404:
--------------------------------------
    Status: In Progress  (was: Open)

> JMX Reporter broken using spark package
> ---------------------------------------
>
>                 Key: HUDI-2404
>                 URL: https://issues.apache.org/jira/browse/HUDI-2404
>             Project: Apache Hudi
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Sarah Witt
>            Priority: Critical
>              Labels: pull-request-available, sev:critical
>             Fix For: 0.10.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I am trying to enable JMX monitoring when using Hudi with Spark.
> I followed the Spark Quickstart tutorial [https://hudi.apache.org/docs/quick-start-guide] and when I added the options to enable JMX monitoring in the insert data stage [https://hudi.apache.org/docs/quick-start-guide#insert-data], I get the following error: `java.lang.NoClassDefFoundError: org/apache/hudi/com/codahale/metrics/jmx/JmxReporter`
>  
> code used:
> df.write.format("hudi").
>   options(getQuickstartWriteConfigs).
>   option("hoodie.metrics.on", "true").
>   option("hoodie.metrics.reporter.type", "JMX").
>   option("hoodie.metrics.reporter.port", "9889").
>   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
>   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
>   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
>   option(TABLE_NAME, tableName).
>   mode(Append).
>   save(basePath)
>  
>  
> metrics-jmx is a new package (not included in metrics-core: [https://metrics.dropwizard.io/4.2.0/about/release-notes.html]), so I believe it should be included in the pom.xml here: [https://github.com/apache/hudi/blob/master/packaging/hudi-spark-bundle/pom.xml#L92-L93]. When I updated my local pom.xml to include metrics-jmx and repeated the setup I was able to successfully enable JMX. The issue seems to happen for flink as well. I can submit a PR if that is easier, too. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)