[jira] [Commented] (BAHIR-233) Add SNS message support for SQS structured streaming connector

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BAHIR-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248441#comment-17248441 ] 

ASF subversion and git services commented on BAHIR-233:
-------------------------------------------------------

Commit 62df1108145ee0305c5c7416a7dadeae5930aab8 in bahir's branch refs/heads/master from DmitryGrb
[ https://gitbox.apache.org/repos/asf?p=bahir.git;h=62df110 ]

[BAHIR-233] Add SNS message support for SQS streaming source (#97)

Added messageWrapper option for SQS streaming connector
which says if this is pure s3 notification event or it is coming
from SNS topic

> Add SNS message support for SQS structured streaming connector
> --------------------------------------------------------------
>
>                 Key: BAHIR-233
>                 URL: https://issues.apache.org/jira/browse/BAHIR-233
>             Project: Bahir
>          Issue Type: New Feature
>          Components: Spark Structured Streaming Connectors
>            Reporter: Dmitry Gorbatsevich
>            Priority: Major
>
> h3. Motivation 
> Current implementation of SQS streaming connector handles the following "route" of the s3 notification event: 
> 1. S3 -> SQS -> Spark 
> This approach works just fine until you need to have multiple listeners (consumers) for the same S3 path. In case multiple applications require to listen and process same S3 path the following approach is recommended: 
> 2. S3 -> SNS -> SQS -> Spark 
> In this case we can route messages from 1 SNS topic to multiple different SQS queues. This enables an ability to listen same S3 path for multiple applications Using approach #2, original S3 notification is wrapped into SNS message and then delivered to the SQS queue. (link to the [AWS docs|https://docs.aws.amazon.com/sns/latest/dg/sns-message-and-json-formats.html] describing SNS message format) 
> To extract original S3 event from SNS message one need to look at "Message" field in json document.  
> h4. Proposed approach 
> # Add option to the s3-sqs connector: "messageWrapper" 
> # It can be 'None' or 'SNS' 
> # Default value is 'None' 
>  
> In case if 'SNS' is specified – "unwrap" original s3 notification event from SNS message and continue processing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)