[jira] [Created] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

["Mark Khaitman (Jira)" <ji...@apache.org>]

Mark Khaitman created SPARK-38881:
-------------------------------------

             Summary: PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs
                 Key: SPARK-38881
                 URL: https://issues.apache.org/jira/browse/SPARK-38881
             Project: Spark
          Issue Type: Improvement
          Components: DStreams, Input/Output, PySpark
    Affects Versions: 3.2.1
            Reporter: Mark Khaitman


This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was merged as part of Spark 3.0.0

This change is desirable as it further exposes the metricsLevel config parameter that was added for the Scala/Java Spark APIs when working with the Kinesis Streaming integration, and makes it available to the PySpark API as well.

This change passes all tests, and local testing was done with a development Kinesis stream in AWS, in order to confirm that metrics were no longer being reported to CloudWatch after specifying MetricsLevel.NONE in the PySpark Kinesis streaming context creation, and also worked as it does today when leaving the MetricsLevel parameter out, which would result in a default of DETAILED, with CloudWatch metrics appearing again.

I plan to open the PR from my forked repo shortly for further discussion if required.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org