You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2025/03/26 19:00:17 UTC

[PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

mxm opened a new pull request, #26357:
URL: https://github.com/apache/flink/pull/26357

   KryoSerializer loads FlinkScalaKryoInstantiator via [Reflection](https://github.com/apache/flink/blob/cc017da9ae12ea58d473d730a84d39440ef928a3/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java#L472) which configures Kryo to use StdInstantiatorStrategy. After FLINK-3154 removed the Twitter Chill library, various types are broken. See https://github.com/EsotericSoftware/kryo/issues/1173 
   
   At first, it seemed that the Kryo update caused this but I can reproduce this also with Kryo 2.24.0 with the Chill library removed.
   
   This change matches the default strategy when the Scala serializers are unavailable: https://github.com/apache/flink/blob/f7a53cede10fa25cb025d5554126a83773ba6bf9/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java#L496
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "cyan-daimao (via GitHub)" <gi...@apache.org>.
cyan-daimao commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3708028524

   "I encountered a similar error in an environment using Iceberg 1.10.1, Flink 2.1.1, and Java 21. The issue was resolved by implementing a custom serializer."
   
   ```java 
           StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
   
           try {
               Class<?> icebergClass = Class.forName("org.apache.iceberg.SerializableByteBufferMap");
   
               ExecutionConfig config = env.getConfig();
   
               SerializerConfigImpl serializerConfig = (SerializerConfigImpl) config.getSerializerConfig();
               serializerConfig.registerTypeWithKryoSerializer(
                       icebergClass,
                       new IcebergMapSerializer() 
               );
   [IcebergInsertDemo.java](https://github.com/user-attachments/files/24421624/IcebergInsertDemo.java)
   
               System.out.println("✅ 成功为 org.apache.iceberg.SerializableByteBufferMap 注册了自定义序列化器。");
   [IcebergInsertDemo.java](https://github.com/user-attachments/files/24421627/IcebergInsertDemo.java)
   
           } catch (ClassNotFoundException e) {
               System.err.println("❌ 错误：找不到类 org.apache.iceberg.SerializableByteBufferMap。请检查您的 Iceberg 依赖版本是否正确。");
               throw e; 
           }
   
           // 3. 创建 TableEnvironment
           StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
   ```
   [IcebergMapSerializer.java](https://github.com/user-attachments/files/24421625/IcebergMapSerializer.java)
   [IcebergInsertDemo.java](https://github.com/user-attachments/files/24421629/IcebergInsertDemo.java)
   
   more: https://juejin.cn/post/7591156262626132003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #26357: [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator
URL: https://github.com/apache/flink/pull/26357


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3570851045

   I think we need to fix some of the custom Scala serializers after this change. The current serializer configuration relies on the default strategy to be `StdInstantiatorStrategy`, which we changed here. I'm going to rebase and revisit this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3018646423

   Hey @michaelparkin! Could you share more details about your use case? When do you run into the serialization issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "neoneo008 (via GitHub)" <gi...@apache.org>.
neoneo008 commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3783025615

   Hey @mxm , Happy New Year. Wanted to see if anything I can help here. Would like to see it complete for [flink 2.3 release](https://cwiki.apache.org/confluence/display/FLINK/2.3+Release). This is currently blocking the mongodb connector to be supported on flink 2.x versions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-2775728390

   Thanks for the review @davidradl!
   
   I briefly looked into adding a test but it proofed a bit difficult due to zero tests for the Scala serializer extensions. The test would have to go into the Scala module or the e2e test module. It isn't possible to test this from flink-core where KryoSerializer resides because the code is dynamically loaded from the table api scala module. 
   
   I'm OoO currently but I'll take a look next week. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-4586068606

   This PR has been closed since it has not had any activity in 120 days. 
   If you feel like this was a mistake, or you would like to continue working on it, 
   please feel free to re-open the PR and ask for a review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-2758935452

   @mxm there are numerous table planner test failures in the CI tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3784307820

   Happy New Year @neoneo008! I'd appreciate if you could look into this. I haven't been able to spend enough time on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26357:
URL: https://github.com/apache/flink/pull/26357#discussion_r2556357235


##########
flink-table/flink-table-api-scala/src/main/scala/org/apache/flink/table/api/runtime/types/FlinkScalaKryoInstantiator.scala:
##########
@@ -59,7 +61,8 @@ class FlinkScalaKryoInstantiator {
   def newKryo: Kryo = {
     val k = new Kryo
     k.setRegistrationRequired(false)
-    k.setInstantiatorStrategy(new org.objenesis.strategy.StdInstantiatorStrategy)
+    val strategy = new DefaultInstantiatorStrategy(new StdInstantiatorStrategy)

Review Comment:
   is it possible to provide a unit test for this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26357:
URL: https://github.com/apache/flink/pull/26357#discussion_r2556357235


##########
flink-table/flink-table-api-scala/src/main/scala/org/apache/flink/table/api/runtime/types/FlinkScalaKryoInstantiator.scala:
##########
@@ -59,7 +61,8 @@ class FlinkScalaKryoInstantiator {
   def newKryo: Kryo = {
     val k = new Kryo
     k.setRegistrationRequired(false)
-    k.setInstantiatorStrategy(new org.objenesis.strategy.StdInstantiatorStrategy)
+    val strategy = new DefaultInstantiatorStrategy(new StdInstantiatorStrategy)

Review Comment:
   is it possible to provide a unit test for this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-4358280253

   This PR is being marked as stale since it has not had any activity in the last 90 days. 
   If you would like to keep this PR alive, please leave a comment asking for a review. 
   If the PR has merge conflicts, update it with the latest from the base branch.
   
   If you are having difficulty finding a reviewer, please reach out to the 
   community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
   
   If this PR is no longer valid or desired, please feel free to close it. 
   If no activity occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-2755528057

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "30a132881642773ee6fe6c2147906fff4474367e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "30a132881642773ee6fe6c2147906fff4474367e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 30a132881642773ee6fe6c2147906fff4474367e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3709845903

   Thanks @cyan-daimao! Surely, registering a custom serializer works, but it's not a nice out of the box experience for the user. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-37546] Use DefaultInstantiatorStrategy for FlinkScalaKryoInstantiator [flink]

Posted by "neoneo008 (via GitHub)" <gi...@apache.org>.
neoneo008 commented on PR #26357:
URL: https://github.com/apache/flink/pull/26357#issuecomment-3568231221

   Hi @mxm @davidradl , any update on the PR. 
   
   We are facing similar issue with Mongodb connectors where we get null pointer since the constructor is not called by kyro. PR which has failure in build logs. https://github.com/apache/flink-connector-mongodb/pull/57 
   
   Happy to help on this PR if anything is pending.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org