[GitHub] [spark] nihalpot commented on a diff in pull request #41791: [WIP][SC-134831] POC for MSK IAM Support

["nihalpot (via GitHub)" <gi...@apache.org>]

nihalpot commented on code in PR #41791:
URL: https://github.com/apache/spark/pull/41791#discussion_r1251304094


##########
docs/structured-streaming-kafka-integration.md:
##########
@@ -1171,3 +1171,14 @@ This can be done several ways. One possibility is to provide additional JVM para
         --driver-java-options "-Djava.security.auth.login.config=/path/to/custom_jaas.conf" \
         --conf spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/path/to/custom_jaas.conf \
         ...
+
+### IAM authentication
+
+Managed Streaming for Kafka (MSK) is a fully managed service offered by Amazon Web Services (AWS) that simplifies the deployment, management, and scaling of Apache Kafka clusters. 
+One way to connect to MSK is by using IAM authentication. To facilitate this, Spark already ships with the [aws msk library](https://github.com/aws/aws-msk-iam-auth) by default so no additional dependencies are required.
+To use IAM authentication, you can set the kafka config as in the [library documentation](https://github.com/aws/aws-msk-iam-auth). For example:
+
+    --conf spark.kafka.clusters.${cluster}.kafka.security.protocol=SASL_SSL

Review Comment:
   Yeah I believe that's how we set spark-confs for `spark-submit`. To make it easier to understand, I have added the syntax with `spark.readStream(...)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org