You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ashic Mahtab <as...@live.com> on 2016/11/15 11:29:23 UTC

Straming - Stop when there's no more data

I'm using Spark Streaming to process a large number of files (10s of millions) from a single directory in S3. Using sparkContext.textFile or wholeTextFile takes ages and doesn't do anything. Pointing Structured Streaming to that location seems to work, but  after processing all the input, it waits for more. Is there a way to terminate the Streaming app once all input has been exhausted?