You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Power Liu (Jira)" <ji...@apache.org> on 2022/09/08 13:44:00 UTC
[jira] [Updated] (HUDI-4815) Hudi report metrics to AWS cloudwatch error
[ https://issues.apache.org/jira/browse/HUDI-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Power Liu updated HUDI-4815:
----------------------------
Description:
*HUDI version:* 0.11.1
*Application:* Spark 3.1.3
*Running Environment:* AWS EKS pod (kubernetes version: 1.21)
*HUDI jar:* org.apache.hudi:hudi-spark-3.1-bundle_2.12:0.11.1
*HUDI metric configuration:*
* hoodie.metrics.reporter.type: CLOUDWATCH
* hoodie.metrics.on: true
* hoodie.metrics.cloudwatch.report.period.seconds: 60
* hoodie.metrics.cloudwatch.namespace: (we defined our own namespace with hyphen inside, like xxxx-xxx-xx)
* hoodie.metrics.cloudwatch.metric.prefix: (we defined our own metric prefix with hyphen inside, like xxxx-xx-xx)
* hoodie.metrics.cloudwatch.maxDatumsPerRequest: 20
*It seems some of the metric reported to Cloudwatch successfully but some are not. Is there any configuration need to adjust according to the error below?*
*Error log:*
ERROR [metrics-hudi-cloudWatch-reporter-1-thread-1] CloudWatchReporter:236 - Error reporting metrics to CloudWatch. The data in this CloudWatch request may have been discarded, and not made it to CloudWatch.
java.util.concurrent.ExecutionException: {color:#ff0000}com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID: xxx-xxxx){color}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID: 3eb96da0-d4bc-4687-81a5-0d65c453559a)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2446)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2413)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2402)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2170)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1170)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1164)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
was:
*HUDI version:* 0.11.1
*Application:* Spark 3.1.3
*Running Environment:* AWS EKS pod (kubernetes version: 1.21)
*HUDI jar:* org.apache.hudi:hudi-spark-3.1-bundle_2.12:0.11.1
*HUDI metric configuration:*
* hoodie.metrics.reporter.type: CLOUDWATCH
* hoodie.metrics.on: true
* hoodie.metrics.cloudwatch.report.period.seconds: 60
* hoodie.metrics.cloudwatch.namespace: (we defined our own namespace with hyphen inside, like xxxx-xxx-xx)
* hoodie.metrics.cloudwatch.metric.prefix: (we defined our own metric prefix with hyphen inside, like xxxx-xx-xx)
* hoodie.metrics.cloudwatch.maxDatumsPerRequest: 20
*It seems some of the metric reported to Cloudwatch successfully but some are not. Is there any configuration need to adjust according to the error below?*
*Error log:*
ERROR [metrics-hudi-cloudWatch-reporter-1-thread-1] CloudWatchReporter:236 - Error reporting metrics to CloudWatch. The data in this CloudWatch request may have been discarded, and not made it to CloudWatch.
java.util.concurrent.ExecutionException: {color:#FF0000}com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID: 3eb96da0-d4bc-4687-81a5-0d65c453559a){color}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID: 3eb96da0-d4bc-4687-81a5-0d65c453559a)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2446)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2413)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2402)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2170)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1170)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1164)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
> Hudi report metrics to AWS cloudwatch error
> -------------------------------------------
>
> Key: HUDI-4815
> URL: https://issues.apache.org/jira/browse/HUDI-4815
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Power Liu
> Priority: Major
>
> *HUDI version:* 0.11.1
> *Application:* Spark 3.1.3
> *Running Environment:* AWS EKS pod (kubernetes version: 1.21)
> *HUDI jar:* org.apache.hudi:hudi-spark-3.1-bundle_2.12:0.11.1
> *HUDI metric configuration:*
> * hoodie.metrics.reporter.type: CLOUDWATCH
> * hoodie.metrics.on: true
> * hoodie.metrics.cloudwatch.report.period.seconds: 60
> * hoodie.metrics.cloudwatch.namespace: (we defined our own namespace with hyphen inside, like xxxx-xxx-xx)
> * hoodie.metrics.cloudwatch.metric.prefix: (we defined our own metric prefix with hyphen inside, like xxxx-xx-xx)
> * hoodie.metrics.cloudwatch.maxDatumsPerRequest: 20
> *It seems some of the metric reported to Cloudwatch successfully but some are not. Is there any configuration need to adjust according to the error below?*
> *Error log:*
> ERROR [metrics-hudi-cloudWatch-reporter-1-thread-1] CloudWatchReporter:236 - Error reporting metrics to CloudWatch. The data in this CloudWatch request may have been discarded, and not made it to CloudWatch.
> java.util.concurrent.ExecutionException: {color:#ff0000}com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID: xxx-xxxx){color}
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
> at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
> at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
> at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID: 3eb96da0-d4bc-4687-81a5-0d65c453559a)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
> at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2446)
> at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2413)
> at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2402)
> at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2170)
> at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1170)
> at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1164)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 common frames omitted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)