You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/07 15:06:09 UTC
[GitHub] [incubator-hudi] jvaesteves opened a new issue #1599: [SUPPORT] Cannot retrieve Graphite metrics
jvaesteves opened a new issue #1599:
URL: https://github.com/apache/incubator-hudi/issues/1599
Hello everyone, I am currently benckmarking Hudi against other solutions for Spark Streaming in terms of time and bytes used. For other solution, I am extracting **durationMs** and **stateOperators** from MicroBatchExecution log, but for Hudi, stateOperators comes empty.
I read at the documentation about the integration with Graphite, but when I set the parameters for Hudi to send metrics to the same location (I tried for localhost or for an external IP) of Spark Metrics (which worked), I started to get errors after the second first microbatch like:
```
20/05/06 11:49:25 ERROR Metrics: Failed to send metrics:
java.lang.IllegalArgumentException: A metric named kafka_hudi.finalize.duration already exists
at org.apache.hudi.com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:97)
at org.apache.hudi.metrics.Metrics.registerGauge(Metrics.java:83)
at org.apache.hudi.metrics.HoodieMetrics.updateFinalizeWriteMetrics(HoodieMetrics.java:177)
at org.apache.hudi.HoodieWriteClient.lambda$finalizeWrite$14(HoodieWriteClient.java:1233)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.HoodieWriteClient.finalizeWrite(HoodieWriteClient.java:1231)
at org.apache.hudi.HoodieWriteClient.commit(HoodieWriteClient.java:497)
at org.apache.hudi.HoodieWriteClient.commit(HoodieWriteClient.java:479)
at org.apache.hudi.HoodieWriteClient.commit(HoodieWriteClient.java:470)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:152)
at org.apache.hudi.HoodieStreamingSink$$anonfun$1$$anonfun$2.apply(HoodieStreamingSink.scala:51)
at org.apache.hudi.HoodieStreamingSink$$anonfun$1$$anonfun$2.apply(HoodieStreamingSink.scala:51)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.hudi.HoodieStreamingSink$$anonfun$1.apply(HoodieStreamingSink.scala:50)
at org.apache.hudi.HoodieStreamingSink$$anonfun$1.apply(HoodieStreamingSink.scala:50)
at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:114)
at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:49)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:84)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:165)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
```
Even though it appears that it successfully sent at least once, I was never able to retrieve these data. Also, for the external IP, I have successfully opened a connection for the address using telnet, so I know that its not a security group issue.
So, what I want to know is if Hudi push metrics to and endpoint like Spark, or if it just exposes those metrics for another to pul. And what could I have done wrong for this application?
Here is my config object for Hudi:
```scala
val hudiOptions = Map[String,String](
HoodieWriteConfig.TABLE_NAME -> hudiTableName,
DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY -> "key",
DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY -> "dt",
DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY -> "timestamp",
DataSourceWriteOptions.OPERATION_OPT_KEY -> "insert",
DataSourceWriteOptions.INSERT_DROP_DUPS_OPT_KEY -> "true",
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED_PROP -> "1",
HoodieMetricsConfig.METRICS_ON -> "true",
HoodieMetricsConfig.METRICS_REPORTER_TYPE -> "GRAPHITE",
HoodieMetricsConfig.GRAPHITE_SERVER_HOST -> "10.115.52.63",
HoodieMetricsConfig.GRAPHITE_SERVER_PORT -> "32683",
HoodieMetricsConfig.GRAPHITE_METRIC_PREFIX -> "hudi"
)
```
Environment Description
- Hudi version: 0.5.0
- Spark version : 2.4.4
- Hive version : 2.3.6
- Hadoop version : Amazon 2.8.5 (emr-5.29.0)
- Storage (HDFS/S3/GCS..) : S3
- Running on Docker? (yes/no) : No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] bvaradar commented on issue #1599: [SUPPORT] Cannot retrieve Graphite metrics
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1599:
URL: https://github.com/apache/incubator-hudi/issues/1599#issuecomment-626310755
Thanks @jvaesteves. Looking forward to your contribution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] leesf commented on issue #1599: [SUPPORT] Cannot retrieve Graphite metrics
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1599:
URL: https://github.com/apache/incubator-hudi/issues/1599#issuecomment-625801473
> Regarding the exception that you mentioned, I agree this is a problem when using Hudi in continuous mode. The metric names are always created in the same fashion and hence after the first run, it starts logging IllegalArgumentExceptions. We need to fix this. May be you could file a jira and contribute the fix back? :)
>
> Regarding not being able to retrieve data at all, does it have anything to do with graphite retention period? Can you check if it is configured properly?
+1 it is a bug and others encountered too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] jvaesteves commented on issue #1599: [SUPPORT] Cannot retrieve Graphite metrics
Posted by GitBox <gi...@apache.org>.
jvaesteves commented on issue #1599:
URL: https://github.com/apache/incubator-hudi/issues/1599#issuecomment-625875000
As requests, I created the issue on Jira: https://issues.apache.org/jira/browse/HUDI-867
I will try to help on this issue this weekend :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1599: [SUPPORT] Cannot retrieve Graphite metrics
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1599:
URL: https://github.com/apache/incubator-hudi/issues/1599#issuecomment-625725108
Regarding the exception that you mentioned, I agree this is a problem when using Hudi in continuous mode. The metric names are always created in the same fashion and hence after the first run, it starts logging IllegalArgumentExceptions. We need to fix this. May be you could file a jira and contribute the fix back? :)
Regarding not being able to retrieve data at all, does it have anything to do with graphite retention period? Can you check if it is configured properly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org