You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "llx206 (GitHub)" <gi...@apache.org> on 2018/12/04 07:08:31 UTC

[GitHub] [incubator-dubbo] llx206 opened issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

https://github.com/apache/incubator-dubbo/issues/2179#issue-347287850

传输的数据较大，想使用kryo序列化，来提高性能。请问，在最新的2.6.5版本下，还是会出现
![image](https://user-images.githubusercontent.com/39985288/49424678-4a839700-f7d6-11e8-8230-815d0555edb1.png)


请问大神们，这个问题要如何解决呢，谢谢！！！！



[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] cvictory commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "cvictory (GitHub)" <gi...@apache.org>.
I find the how it is happen.  I will commit a pr to complete the kryo
serialization.
But if you want to control the registry class , there will be some constraint
to use.

llx206 <no...@github.com> 于2018年12月6日周四 下午5:31写道：

> @eoc2015 <https://github.com/eoc2015> ,是要修改dubbo的代码，才能解决吗？
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-dubbo/issues/2881#issuecomment-444807244>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AD5mbFEyKQDI5IDThVzoS9vXVpbHl20Dks5u2OPkgaJpZM4ZADEY>
> .
>


-- 

Best Regard!
cvitory


[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] cvictory commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "cvictory (GitHub)" <gi...@apache.org>.
kryo类型查找会有两种形式，一种是根据类名，一种是根据id和类名的映射。
如果只是根据类名进行映射，不会出现问题，性能稍差；如果想根据id和类名的映射，性能较好，但是id和class的关系，必须保证在server端和client端的顺序一致。
在dubbo中，当你使用optimizer="org.apache.dubbo.benchmark.serialize.SerializationOptimizerImpl"
配置时，就会针对SerializationOptimizerImpl中定义的class开启id和class映射。代码如下：
*```*

public Collection<Class> getSerializableClasses() {
    return Arrays.asList(User.class, Page.class, UserService.class);
}

*```*
*整个调用出现问题的原理可以看下图。*
![image](https://user-images.githubusercontent.com/4089452/49933990-2297e000-ff08-11e8-8861-fd5931d1fc5a.png)


所以解决办法有两种：
1.不要启用这个优化，也就是不要设置optimizer。
2.dubbo代码进行了初步的优化，进行了初步的测试，可以运行。原理如图：

![image](https://user-images.githubusercontent.com/4089452/49934014-33e0ec80-ff08-11e8-9f04-a9f8104946ce.png)


原来的接口里有一个全局的映射表，这个逻辑不改变的前提下；针对部分接口维度的优化，可以新建一个id和class列表。这样可以实现功能，但是同时带来了新建和维护这个列表的成本。

victory <sh...@gmail.com> 于2018年12月12日周三 下午3:04写道：

> I find the how it is happen.  I will commit a pr to complete the kryo
> serialization.
> But if you want to control the registry class , there will be some constraint
> to use.
>
> llx206 <no...@github.com> 于2018年12月6日周四 下午5:31写道：
>
>> @eoc2015 <https://github.com/eoc2015> ,是要修改dubbo的代码，才能解决吗？
>>
>> —
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/apache/incubator-dubbo/issues/2881#issuecomment-444807244>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AD5mbFEyKQDI5IDThVzoS9vXVpbHl20Dks5u2OPkgaJpZM4ZADEY>
>> .
>>
>
>
> --
>
> Best Regard!
> cvitory
>


-- 

Best Regard!
cvitory


[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] llx206 commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "llx206 (GitHub)" <gi...@apache.org>.
@cvictory 非常感谢你的再次解答。同时，也非常期待你们的新版本能用更快速的方式实现，期待新版本。谢谢！！！

[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] llx206 commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "llx206 (GitHub)" <gi...@apache.org>.
@cvictory ，非常感谢你细致的回复。还有两个问题再请教下。
1.不要启用这个优化，也就是不要设置optimizer。
问：不启用优化，相比hessian2序列化协议，kryo的性能如何呢？
2.dubbo代码进行了初步的优化，进行了初步的测试，可以运行
问：针对部分接口维度的优化，新建一个ID和class列表，这个会在dubbo的哪个版本加入呢？

期待你的再次回复，谢谢！！

[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] eoc2015 commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "eoc2015 (GitHub)" <gi...@apache.org>.
感觉这个可以参考：https://www.2cto.com/kf/201805/743957.html

[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] cvictory commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "cvictory (GitHub)" <gi...@apache.org>.
抱歉，上周休假了，网络实在是惨不忍睹。
问题1，我们没做过测试，部分数据来源于网络：https://x-rip.iteye.com/blog/1555293 。
同时强调一点，如果不用注册方式的话，尽量让报名和类名短，来自于：https://github.com/EsotericSoftware/kryo#optional-registration 。

问题2，预计要到2.6.6版本中支持。 我还在考虑是否可以考虑使用一种比较快速的方式来解决。 你可以在上面看到我提交的pr。

----
Sorry, the vacation last week, the network is really terrible.
Question 1, we have not tested, some of the data comes from the network: https://x-rip.iteye.com/blog/1555293.
At the same time, it is emphasized that if you do not need to register, try to make the registration and class name short, from: https://github.com/EsotericSoftware/kryo#optional-registration.

Question 2 is expected to be supported in version 2.6.6. I am still considering whether I can consider using a faster way to solve it. You can see the pr I submitted above.

[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] cvictory commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "cvictory (GitHub)" <gi...@apache.org>.
kryo类型查找会有两种形式，一种是根据类名，一种是根据id和类名的映射。
如果只是根据类名进行映射，不会出现问题，性能稍差；如果想根据id和类名的映射，性能较好，但是id和class的关系，必须保证在server端和client端的顺序一致。
在dubbo中，当你使用optimizer="org.apache.dubbo.benchmark.serialize.SerializationOptimizerImpl"
配置时，就会针对SerializationOptimizerImpl中定义的class开启id和class映射。代码如下：
*```*

public Collection<Class> getSerializableClasses() {
    return Arrays.asList(User.class, Page.class, UserService.class);
}

*```*
*整个调用出现问题的原理可以看下图。*

[image: image.png]

所以解决办法有两种：
1.不要启用这个优化，也就是不要设置optimizer。
2.dubbo代码进行了初步的优化，进行了初步的测试，可以运行。原理如图：
[image: image.png]


原来的接口里有一个全局的映射表，这个逻辑不改变的前提下；针对部分接口维度的优化，可以新建一个id和class列表。这样可以实现功能，但是同时带来了新建和维护这个列表的成本。

victory <sh...@gmail.com> 于2018年12月12日周三 下午3:04写道：

> I find the how it is happen.  I will commit a pr to complete the kryo
> serialization.
> But if you want to control the registry class , there will be some constraint
> to use.
>
> llx206 <no...@github.com> 于2018年12月6日周四 下午5:31写道：
>
>> @eoc2015 <https://github.com/eoc2015> ,是要修改dubbo的代码，才能解决吗？
>>
>> —
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/apache/incubator-dubbo/issues/2881#issuecomment-444807244>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AD5mbFEyKQDI5IDThVzoS9vXVpbHl20Dks5u2OPkgaJpZM4ZADEY>
>> .
>>
>
>
> --
>
> Best Regard!
> cvitory
>


-- 

Best Regard!
cvitory


[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] llx206 commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "llx206 (GitHub)" <gi...@apache.org>.
@eoc2015 ,是要修改dubbo的代码，才能解决吗？

[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org

[GitHub] [incubator-dubbo] cvictory commented on issue #2881: Kryo serialization fails（服务既是生产者又是消费者出现问题）

Posted by "cvictory (GitHub)" <gi...@apache.org>.
kryo类型查找会有两种形式，一种是根据类名，一种是根据id和类名的映射。
如果只是根据类名进行映射，不会出现问题，性能稍差；如果想根据id和类名的映射，性能较好，但是id和class的关系，必须保证在server端和client端的顺序一致。
在dubbo中，当你使用optimizer="org.apache.dubbo.benchmark.serialize.SerializationOptimizerImpl"
配置时，就会针对SerializationOptimizerImpl中定义的class开启id和class映射。代码如下：
*```*

public Collection<Class> getSerializableClasses() {
    return Arrays.asList(User.class, Page.class, UserService.class);
}

*```*
*整个调用出现问题的原理可以看下图。*

[image: image.png]
![image](https://user-images.githubusercontent.com/4089452/49933932-fda36d00-ff07-11e8-9b24-5beb3321b6fd.png)


所以解决办法有两种：
1.不要启用这个优化，也就是不要设置optimizer。
2.dubbo代码进行了初步的优化，进行了初步的测试，可以运行。原理如图：

![Uploading image.png…]()


原来的接口里有一个全局的映射表，这个逻辑不改变的前提下；针对部分接口维度的优化，可以新建一个id和class列表。这样可以实现功能，但是同时带来了新建和维护这个列表的成本。

victory <sh...@gmail.com> 于2018年12月12日周三 下午3:04写道：

> I find the how it is happen.  I will commit a pr to complete the kryo
> serialization.
> But if you want to control the registry class , there will be some constraint
> to use.
>
> llx206 <no...@github.com> 于2018年12月6日周四 下午5:31写道：
>
>> @eoc2015 <https://github.com/eoc2015> ,是要修改dubbo的代码，才能解决吗？
>>
>> —
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/apache/incubator-dubbo/issues/2881#issuecomment-444807244>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AD5mbFEyKQDI5IDThVzoS9vXVpbHl20Dks5u2OPkgaJpZM4ZADEY>
>> .
>>
>
>
> --
>
> Best Regard!
> cvitory
>


-- 

Best Regard!
cvitory


[ Full content available at: https://github.com/apache/incubator-dubbo/issues/2881 ]
This message was relayed via gitbox.apache.org for notifications@dubbo.apache.org