You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/01/25 10:07:43 UTC

[GitHub] gaborgsomogyi commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query

gaborgsomogyi commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
URL: https://github.com/apache/spark/pull/22952#issuecomment-457522126
 
 
   @HeartSaVioR @zsxwing 
   Just one addition to the rename behaviour considering the mentioned scenario:
   > I feel the current behavior of rename is a bit weird. Let's say the source dir is /a/b/c and sourceArchiveDir is /a/b/c/_archive, if the file to rename is /a/b/c/d=1/x.txt, the final path will be /a/b/c/_archive/a/b/c/d=1/x.txt. Is there a special reason? Can we try to make it be /a/b/c/_archive/d=1/x.txt?
   
   Here is a potential weird scenario:
   sourceGlob = /a/b/c*/d*
   sourceArchiveDir = /a/b/c/_archive
   file1 = /a/b/c/d=1/x.txt
   file2 = /a/b/cc/d=1/x.txt
   
   End result:
   /a/b/c/_archive/d=1/x.txt
   /a/b/c/_archive/a/b/cc/d=1/x.txt
   
   As a user I would be surprised to see such directory structure. Full path is more intuitive to me as well.
   The other option would be to block directory overlap but it's not feasible in a fast way because of glob (please see discussion).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org