[jira] [Commented] (SPARK-30281) 'archive' option in FileStreamSource misses to consider partitioned and recursive option

["Jungtaek Lim (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-30281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997905#comment-16997905 ] 

Jungtaek Lim commented on SPARK-30281:
--------------------------------------

Will submit a PR soon.

> 'archive' option in FileStreamSource misses to consider partitioned and recursive option
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-30281
>                 URL: https://issues.apache.org/jira/browse/SPARK-30281
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> Cleanup option for FileStreamSource is introduced in SPARK-20568.
> To simplify the condition of verifying archive path, it took the fact that FileStreamSource reads the files where these files meet one of conditions: 1) parent directory matches the source pattern 2) the file itself matches the source pattern.
> We found there're other cases during post-hoc review which invalidate above fact: partitioned, and recursive option. With these options, FileStreamSource can read the arbitrary files in subdirectories which match the source pattern, so simply checking the depth of archive path doesn't work.
> We need to restore the path check logic, though it would be not easy to explain to end users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org