You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Maximilian Michels (Jira)" <ji...@apache.org> on 2020/01/06 18:42:01 UTC
[jira] [Resolved] (BEAM-8577) FileSystems may have not be
initialized during ResourceId deserialization
[ https://issues.apache.org/jira/browse/BEAM-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maximilian Michels resolved BEAM-8577.
--------------------------------------
Fix Version/s: 2.19.0
Resolution: Fixed
> FileSystems may have not be initialized during ResourceId deserialization
> -------------------------------------------------------------------------
>
> Key: BEAM-8577
> URL: https://issues.apache.org/jira/browse/BEAM-8577
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.16.0
> Reporter: David Morávek
> Assignee: David Morávek
> Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> - FileSystems use static registration using *FileSystems#setDefaultPipelineOptions* method.
> - *#setDefaultPipelineOptions* is called either when deserializaing SerializablePipelineOptions or during opening of various beam operators.
> - *FileIO#matchAll* is expanded using *Reshuffle.viaRandomKey()*.
> - Reshuffle is implemented using *.rebalance*, that doesn't have a "RichFunction" lifecycle, so we need to find another way to register FileSystems, as the deserialization may happen before other "rich operators" get executed on particular task manager.
> This results in random pipeline fails as the task assignment is not deterministic.
> We can workaround this, by registering FileSystems during coder deserialization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)