[jira] [Commented] (BEAM-10295) FileBasedSink: allow setting temp directory provider per dynamic destination

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182016#comment-17182016 ] 

Beam JIRA Bot commented on BEAM-10295:
--------------------------------------

This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3.

Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean.


> FileBasedSink: allow setting temp directory provider per dynamic destination
> ----------------------------------------------------------------------------
>
>                 Key: BEAM-10295
>                 URL: https://issues.apache.org/jira/browse/BEAM-10295
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-hadoop-file-system, sdk-java-core
>            Reporter: David Janicek
>            Priority: P2
>              Labels: stale-P2
>
> Dynamic file destinations allow value-dependent writes in FileBasedSink. When using hadoop file system this means user can write some values to destination at *cluster-A* and some values to destination at *cluster-B*.
> Since BEAM-7613 was fixed this works fine until the *moveToOutputFiles* method is called. This method internally calls *FileSystems.rename* which obviously requires that source files (temporary files) and target files (resolved by dynamic destination's function) are on the same cluster. But the temp directory provider can be set only one per file sink.
> This could be fixed by adding some kind of *getTempDirectoryProvider* method into dynamic destinations (e.g. into *DefaultFilenamePolicy.Params*).
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)