You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jerome Gagnon (JIRA)" <ji...@apache.org> on 2019/04/09 19:03:00 UTC
[jira] [Updated] (SPARK-27420) KinesisInputDStream should expose a
way to disable CloudWatch metrics
[ https://issues.apache.org/jira/browse/SPARK-27420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Gagnon updated SPARK-27420:
----------------------------------
Priority: Major (was: Minor)
> KinesisInputDStream should expose a way to disable CloudWatch metrics
> ---------------------------------------------------------------------
>
> Key: SPARK-27420
> URL: https://issues.apache.org/jira/browse/SPARK-27420
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Input/Output
> Affects Versions: 2.3.3
> Reporter: Jerome Gagnon
> Priority: Major
>
> KinesisInputDStream currently does not provide a way to disable CloudWatch metrics push. Kinesis client library (KCL) which is used under the hood provide the ability through `withMetrics` methods.
> To make things worse the default level is "DETAILED" which pushes 10s of metrics every 10 seconds. When dealing with multiple streaming jobs this add up pretty quickly, leading to thousands of dollar in cost.
> Exposing a way to disable/set the proper level of monitoring is critical to us. We had to send invalid credentials and suppress log as a less-than-ideal workaround : see [https://stackoverflow.com/questions/41811039/disable-cloudwatch-for-aws-kinesis-at-spark-streaming/55599002#55599002]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org