[GitHub] [spark] etspaceman opened a new pull request #24801: dynamoDBEndpointUrl and cloudWatchMetricsLevel for Kinesis

[GitBox <gi...@apache.org>]

etspaceman opened a new pull request #24801: dynamoDBEndpointUrl and cloudWatchMetricsLevel for Kinesis
URL: https://github.com/apache/spark/pull/24801
 
 
   ## What changes were proposed in this pull request?
   
   I was researching getting Spark’s Kinesis integration running locally against `localstack`. We found this issue, and it creates a complication: https://github.com/localstack/localstack/issues/677
   
   Effectively, we need to be able to redirect calls for Kinesis, DynamoDB and Cloudwatch in order for the KCL to properly use the `localstack` infrastructure. We have successfully done this with the KCL (both 1.x and 2.x), but with Spark’s integration we are unable to configure DynamoDB and Cloudwatch’s endpoints:
   
   https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162
   
   This PR adds optional configuration values to the interfaces for dynamoDBEndpointUrl and cloudWatchMetricsLevel. 
   
   Why cloudWatchMetricsLevel instead of cloudWatchEndpointUrl? Because the 1.x version of the KCL does not expose a means of configuring the cloudWatchEndpointUrl. Localstack users can instead disable metrics entirely by setting the cloudWatchMetricsLevel to Some(MetricsLevel.NONE)
   
   ## How was this patch tested?
   
   Existing unit tests were expanded to check that these values were set.
   
   Please review https://spark.apache.org/contributing.html before opening a pull request.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org