You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/01/25 10:07:43 UTC
[GitHub] gaborgsomogyi commented on issue #22952: [SPARK-20568][SS] Provide
option to clean up completed files in streaming query
gaborgsomogyi commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
URL: https://github.com/apache/spark/pull/22952#issuecomment-457522126
@HeartSaVioR @zsxwing
Just one addition to the rename behaviour considering the mentioned scenario:
> I feel the current behavior of rename is a bit weird. Let's say the source dir is /a/b/c and sourceArchiveDir is /a/b/c/_archive, if the file to rename is /a/b/c/d=1/x.txt, the final path will be /a/b/c/_archive/a/b/c/d=1/x.txt. Is there a special reason? Can we try to make it be /a/b/c/_archive/d=1/x.txt?
Here is a potential weird scenario:
sourceGlob = /a/b/c*/d*
sourceArchiveDir = /a/b/c/_archive
file1 = /a/b/c/d=1/x.txt
file2 = /a/b/cc/d=1/x.txt
End result:
/a/b/c/_archive/d=1/x.txt
/a/b/c/_archive/a/b/cc/d=1/x.txt
As a user I would be surprised to see such directory structure. Full path is more intuitive to me as well.
The other option would be to block directory overlap but it's not feasible in a fast way because of glob (please see discussion).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org