[jira] [Updated] (SPARK-48802) FileStreamSource: maxCachedFiles set to 0 causes empty batches

["kevin nacios (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-48802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

kevin nacios updated SPARK-48802:
---------------------------------
    Component/s: Structured Streaming
                     (was: SS)

> FileStreamSource: maxCachedFiles set to 0 causes empty batches
> --------------------------------------------------------------
>
>                 Key: SPARK-48802
>                 URL: https://issues.apache.org/jira/browse/SPARK-48802
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.0.0
>         Environment: Tested with Spark 4.0 RC
>            Reporter: kevin nacios
>            Priority: Minor
>
> While testing changes from https://issues.apache.org/jira/browse/SPARK-44924 it was discovered that setting {{maxCachedFiles}} to 0 to try to force each batch to perform a full listing was inadvertently causing every other batch to trigger with no files being processed.  
> In this case, the list of {{unselectedFiles}} from a file listing gets 0 files taken from it and assigned to {{unreadFiles}} as an empty list, which is re-used in the next batch and only checked if {{unreadFiles}} is null.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org