[GitHub] [bahir] abhishekd0907 commented on a change in pull request #97: [BAHIR-233] Add SNS message support for SQS streaming source

[GitBox <gi...@apache.org>]

abhishekd0907 commented on a change in pull request #97:
URL: https://github.com/apache/bahir/pull/97#discussion_r449407090



##########
File path: sql-streaming-sqs/src/main/scala/org/apache/spark/sql/streaming/sqs/SqsClient.scala
##########
@@ -131,13 +131,24 @@ class SqsClient(sourceOptions: SqsSourceOptions,
     }
   }
 
+  private def extractS3Message(parsedBody: JValue): JValue = {
+    implicit val formats = DefaultFormats
+    sourceOptions.messageWrapper match {
+      case sourceOptions.S3MessageWrapper.None => parsedBody
+      case sourceOptions.S3MessageWrapper.SNS => parse((parsedBody \ "Message").extract[String])

Review comment:
       Will this also throw Mapping Exception when it encounters a message in different format? Say, `S3MessageWrapper` is set to SNS but SQS queue has some message which comes directly from S3 and not via SNS

##########
File path: sql-streaming-sqs/README.md
##########
@@ -63,6 +63,7 @@ shouldSortFiles|true|whether to sort files based on timestamp while listing them
 useInstanceProfileCredentials|false|Whether to use EC2 instance profile credentials for connecting to Amazon SQS
 maxFilesPerTrigger|no default value|maximum number of files to process in a microbatch
 maxFileAge|7d|Maximum age of a file that can be found in this directory
+messageWrapper|None|'None' if SQS contains plain S3 message. 'SNS' if SQS contains S3 notification message which came from SNS

Review comment:
       can you add a small 2-3 lines section in Readme explaining how to use S3-SQS data-source for multiple consumers  by setting `messageWrapper` to SNS? Whatever you've explained in the JIRA should be sufficient.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org