You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/08/22 15:47:40 UTC

[GitHub] [flink] zentol opened a new pull request, #20661: [FLINK-29030][core] Improve messaging around Kryo & schema evolution

zentol opened a new pull request, #20661:
URL: https://github.com/apache/flink/pull/20661

   The type extractors will now _always_ log a message if a tuple/pojo contains a generic type. Additionally we now also give a hint that schema evolution may be affected, and finally clarify the schema evolution docs about Kryo blocking it for types contained in serialized data structures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #20661: [FLINK-29030][core] Improve messaging around Kryo & schema evolution

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20661:
URL: https://github.com/apache/flink/pull/20661#issuecomment-1222553172

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "88e2d7517dfdff8b9b2d25fc488034ce8f9c96e1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "88e2d7517dfdff8b9b2d25fc488034ce8f9c96e1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 88e2d7517dfdff8b9b2d25fc488034ce8f9c96e1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] alpinegizmo commented on a diff in pull request #20661: [FLINK-29030][core] Improve messaging around Kryo & schema evolution

Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on code in PR #20661:
URL: https://github.com/apache/flink/pull/20661#discussion_r953225902


##########
docs/content/docs/dev/datastream/fault-tolerance/serialization/schema_evolution.md:
##########
@@ -125,4 +125,10 @@ Additionally, the RocksDB state backend relies on binary object identity, rather
 
 When Kryo is used, there is no possibility for the framework to verify if any incompatible changes have been made.
 
+{{< hint warning >}}
+This means that if a data-structure containing a given type is serialized via Kryo, then that contained type can **not** undergo schema evolution.
+
+For example, if a POJO contains a `List<SomeOtherPojo>`, then the `List` _and_ it's contents are serialized via Kryo and schema evolution is **not** supported for `SomeOtherPojo`.

Review Comment:
   ```suggestion
   For example, if a POJO contains a `List<SomeOtherPojo>`, then the `List` _and_ its contents are serialized via Kryo and schema evolution is **not** supported for `SomeOtherPojo`.
   ```



##########
docs/content.zh/docs/dev/datastream/fault-tolerance/serialization/schema_evolution.md:
##########
@@ -95,16 +95,29 @@ Flink 完全支持 Avro 状态类型的升级,只要数据结构的修改是
 
 一个例外是如果新的 Avro 数据 schema 生成的类无法被重定位或者使用了不同的命名空间,在作业恢复时状态数据会被认为是不兼容的。
 
-{{< hint warning >}}
-Schema evolution of keys is not supported.
-{{< /hint >}}
+## Schema Migration Limitations
 
-Example: RocksDB state backend relies on binary objects identity, rather than `hashCode` method implementation. Any changes to the keys object structure could lead to non deterministic behaviour.
+Flink's schema migration has some limitations that are required to ensure correctness. For users that need to work
+around these limitations, and understand them to be safe in their specific use-case, consider using
+a [custom serializer]({{< ref "docs/dev/datastream/fault-tolerance/serialization/custom_serialization" >}}) or the
+[state processor api]({{< ref "docs/libs/state_processor_api" >}}).
 
-{{< hint warning >}}
-**Kryo** cannot be used for schema evolution.
-{{< /hint >}}
+### Schema evolution of keys is not supported.
+
+The structure of a key cannot be migrated as this may lead to non-deterministic behavior.
+For example, if a POJO is used as a key and one field is dropped then there may suddenly be
+multiple separate keys that are now identical. Flink has no way to merge the corresponding values.
+
+Additionally, the RocksDB state backend relies on binary object identity, rather than the `hashCode` method. Any change to the keys' object structure can lead to non-deterministic behavior.
+
+### **Kryo** cannot be used for schema evolution.
 
 When Kryo is used, there is no possibility for the framework to verify if any incompatible changes have been made.
 
+{{< hint warning >}}
+This means that if a data-structure containing a given type is serialized via Kryo, then that contained type can **not** undergo schema evolution.
+
+For example, if a POJO contains a `List<SomeOtherPojo>`, then the `List` _and_ it's contents are serialized via Kryo and schema evolution is **not** supported for `SomeOtherPojo`.

Review Comment:
   ```suggestion
   For example, if a POJO contains a `List<SomeOtherPojo>`, then the `List` _and_ its contents are serialized via Kryo and schema evolution is **not** supported for `SomeOtherPojo`.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] zentol merged pull request #20661: [FLINK-29030][core] Improve messaging around Kryo & schema evolution

Posted by GitBox <gi...@apache.org>.
zentol merged PR #20661:
URL: https://github.com/apache/flink/pull/20661


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org