You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yuan Mei (Jira)" <ji...@apache.org> on 2022/04/01 12:46:00 UTC
[jira] [Updated] (FLINK-26992) PojoSerializer may cause concurrent exception passing directly between threads
[ https://issues.apache.org/jira/browse/FLINK-26992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuan Mei updated FLINK-26992:
-----------------------------
Description:
Extract from FLINK-26835
While investigating this issue, we have found that probably state backends are also using non-thread safe serialisers from different threads.
For example: {{RocksFullSnapshotStrategy#syncPrepareResources}} is passing {{keySerializer}} from the task thread, to the async thread in order to serialize the serializer itself. {{RocksIncrementalSnapshotStrategy.RocksDBIncrementalSnapshotOperation#materializeMetaData}} seems to be doing the same thing. If {{PojoSerializer}} is used as {{keySerializer}} I think this will lead to the same problems as above. Iterating through the {{PojoSerializer#subclassSerializerCache}} from the the async checkpoint thread, while the map can be changed from the task thread. It looks like in all of those places the serializer should have been duplicated ({{{}#duplicate{}}}) before being passed to another thread. Maybe this should happen in {{{}RocksDBSnapshotStrategyBase{}}}. I don't know about other state backends.
> PojoSerializer may cause concurrent exception passing directly between threads
> ------------------------------------------------------------------------------
>
> Key: FLINK-26992
> URL: https://issues.apache.org/jira/browse/FLINK-26992
> Project: Flink
> Issue Type: Bug
> Reporter: Yuan Mei
> Priority: Major
>
> Extract from FLINK-26835
> While investigating this issue, we have found that probably state backends are also using non-thread safe serialisers from different threads.
> For example: {{RocksFullSnapshotStrategy#syncPrepareResources}} is passing {{keySerializer}} from the task thread, to the async thread in order to serialize the serializer itself. {{RocksIncrementalSnapshotStrategy.RocksDBIncrementalSnapshotOperation#materializeMetaData}} seems to be doing the same thing. If {{PojoSerializer}} is used as {{keySerializer}} I think this will lead to the same problems as above. Iterating through the {{PojoSerializer#subclassSerializerCache}} from the the async checkpoint thread, while the map can be changed from the task thread. It looks like in all of those places the serializer should have been duplicated ({{{}#duplicate{}}}) before being passed to another thread. Maybe this should happen in {{{}RocksDBSnapshotStrategyBase{}}}. I don't know about other state backends.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)