[jira] [Resolved] (SPARK-28124) Faster S3 file source with SQS

["Abhishek Dixit (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-28124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhishek Dixit resolved SPARK-28124.
------------------------------------
    Resolution: Feedback Received

As suggested by [~gsomogyi], [~stevel@apache.org], [~zsxwing] I've opened a [pull request|[https://github.com/apache/bahir/pull/91]] in BAHIR for this, so closing this ticket. 

> Faster S3 file source with SQS
> ------------------------------
>
>                 Key: SPARK-28124
>                 URL: https://issues.apache.org/jira/browse/SPARK-28124
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 3.0.0
>            Reporter: Abhishek Dixit
>            Priority: Major
>
> Using FileStreamSource to read files from a S3 bucket has problems both in terms of costs and latency:
>  * *Latency:* Listing all the files in S3 buckets every microbatch can be both slow and resource intensive.
>  * *Costs:* Making List API requests to S3 every microbatch can be costly.
>  The solution is to use Amazon Simple Queue Service (SQS) which lets you find new files written to S3 bucket without the need to list all the files every microbatch.
> S3 buckets can be configured to send notification to an Amazon SQS Queue on Object Create / Object Delete events. For details see AWS documentation here [Configuring S3 Event Notifications|https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html] 
> Spark can leverage this to find new files written to S3 bucket by reading notifications from SQS queue instead of listing files every microbatch.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org