[jira] [Commented] (BEAM-8577) FileSystems may have not be initialized during ResourceId deserialization

["Kenneth Knowles (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976835#comment-16976835 ] 

Kenneth Knowles commented on BEAM-8577:
---------------------------------------

[~mxm] [~davidmoravek] I see a PR under review. Is this a regression? Or is it quick & easy to finish review and cherrypick? Asking because it is tagged for 2.17.0

> FileSystems may have not be initialized during ResourceId deserialization
> -------------------------------------------------------------------------
>
>                 Key: BEAM-8577
>                 URL: https://issues.apache.org/jira/browse/BEAM-8577
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.16.0
>            Reporter: David Moravek
>            Assignee: David Moravek
>            Priority: Major
>             Fix For: 2.17.0
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> - FileSystems use static registration using *FileSystems#setDefaultPipelineOptions* method.
> - *#setDefaultPipelineOptions* is called either when deserializaing SerializablePipelineOptions or during opening of various beam operators. 
> - *FileIO#matchAll* is expanded using *Reshuffle.viaRandomKey()*.
> - Reshuffle is implemented using *.rebalance*, that doesn't have a "RichFunction" lifecycle, so we need to find another way to register FileSystems, as the deserialization may happen before other "rich operators" get executed on particular task manager.
> This results in random pipeline fails as the task assignment is not deterministic.
> We can workaround this, by registering FileSystems during coder deserialization.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)