[GitHub] [beam] damccorm opened a new issue, #20531: FileBasedSink: allow setting temp directory provider per dynamic destination

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #20531:
URL: https://github.com/apache/beam/issues/20531

   Dynamic file destinations allow value-dependent writes in FileBasedSink. When using hadoop file system this means user can write some values to destination at *cluster-A* and some values to destination at *cluster-B*.
   
   Since BEAM-7613 was fixed this works fine until the *moveToOutputFiles* method is called. This method internally calls *FileSystems.rename* which obviously requires that source files (temporary files) and target files (resolved by dynamic destination's function) are on the same cluster. But the temp directory provider can be set only one per file sink.
   
   This could be fixed by adding some kind of *getTempDirectoryProvider* method into dynamic destinations (e.g. into *DefaultFilenamePolicy.Params*).
   
    
   
    
   
   Imported from Jira [BEAM-10295](https://issues.apache.org/jira/browse/BEAM-10295). Original Jira may contain additional context.
   Reported by: davidak09.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org