You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/24 03:01:35 UTC
[GitHub] [rocketmq-schema-registry] MatrixHB opened a new issue, #49: Support cache for SchemaRegistryClient
MatrixHB opened a new issue, #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49
The client obtains the schema through `schemaRegistryClient`. It is necessary to consider adding schema cache on the client side, to avoid too much remote requests for serialization or deserialization.
Cache needs to be designed with reasonable key-value struct and expiration mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] ferrirW commented on issue #49: Support cache for SchemaRegistryClient
Posted by "ferrirW (via GitHub)" <gi...@apache.org>.
ferrirW commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1408130086
> 3. 由于客户端中的getschema方法需要的返回值都是GetSchemaResponse类型，因此我的缓存value也保存这个类型的值。这样的优点在于能够直接返回缓存中的value，但是缺点在于占用更多的空间，并且在register的过程中可能无法更新缓存（因为数据无法拼接成一个GetSchemaResponse）。我暂时没有想到更好的办法
最佳做法应该是在 restservice 中对 response 进行一次拆包和封装，转为类似 SchemaInfo 的类 @humkum 可以看看做一下封装
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] ferrirW commented on issue #49: Support cache for SchemaRegistryClient
Posted by "ferrirW (via GitHub)" <gi...@apache.org>.
ferrirW commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1408128717
> 2. 我理解缓存主要应用于getschema的各个函数中，但由于查询的入参组合种类很多，我可能不得不对于每种组合都进行缓存。例如对于getSchemaByRecordId函数，缓存的key需要设置为SubjectAndId；而在getSchemaBySubjectAndVersion中，缓存的key则需要为SubjectAndVersion，因此就需要两个map来进行分别存储。这种方法的缺点在于会存在一些重复的value，占用的空间更多，因此想和您讨论一下是否有更好的解决方法。
也许我们可以使用一个根 cache 来缓存数据，例如 <Pair<subject, recordId>, GetSchemaResponse>，其他不同的接口仅仅是对其 key 的构造，例如 getSchemaBySubjectAndVersion 中的 version 可以通过雪花算法的逆向与 recordId 进行匹配；
当然现在的重复的占用仅是引用，我认为开销可以接受
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] joeCarf commented on issue #49: Support cache for SchemaRegistryClient
Posted by GitBox <gi...@apache.org>.
joeCarf commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1370587302
> The client obtains the schema through `schemaRegistryClient`. It is necessary to consider adding schema cache on the client side, to avoid too much remote requests for serialization or deserialization.
>
> Cache needs to be designed with reasonable key-value struct and expiration mechanism.
hi, @MatrixHB 我最近在设计客户端缓存，同时也参考了kafka schema registry的实现，但在过程中遇到了一点问题想和您讨论一下 ：
1. 在使用过程中哪些函数的使用频率更高，还是说对于client中的所有函数都需要进行缓存呢
2. 我理解缓存主要应用于getschema的各个函数中，但由于查询的入参组合种类很多，我可能不得不对于每种组合都进行缓存。例如对于getSchemaByRecordId函数，缓存的key需要设置为SubjectAndId；而在getSchemaBySubjectAndVersion中，缓存的key则需要为SubjectAndVersion，因此就需要两个map来进行分别存储。这种方法的缺点在于会存在一些重复的value，占用的空间更多，因此想和您讨论一下是否有更好的解决方法。
3. 由于客户端中的getschema方法需要的返回值都是GetSchemaResponse类型，因此我的缓存value也保存这个类型的值。这样的优点在于能够直接返回缓存中的value，但是缺点在于占用更多的空间，并且在register的过程中可能无法更新缓存（因为数据无法拼接成一个GetSchemaResponse）。我暂时没有想到更好的办法
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] joeCarf commented on issue #49: Support cache for SchemaRegistryClient
Posted by "joeCarf (via GitHub)" <gi...@apache.org>.
joeCarf commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1426698356
> > > 3. 由于客户端中的getschema方法需要的返回值都是GetSchemaResponse类型，因此我的缓存value也保存这个类型的值。这样的优点在于能够直接返回缓存中的value，但是缺点在于占用更多的空间，并且在register的过程中可能无法更新缓存（因为数据无法拼接成一个GetSchemaResponse）。我暂时没有想到更好的办法
> >
> >
> > 最佳做法应该是在 restservice 中对 response 进行一次拆包和封装，转为类似 SchemaInfo 的类 @humkum 可以看看做一下封装
>
> 嗯嗯。我觉得或许不应该直接暴露GetSchemaResponse给用户，这样也方便在客户端进行缓存。如果 @humkum 同学在做这个封装的话，我希望能够一起合作～
如果能有一个统一的封装来返回给用户，而不是直接使用GetSchemaResponse/UpdateSchemaReponse/RegisterSchemaResponse的话，我觉得会大大减轻客户端缓存的空间，操作也会更便捷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] joeCarf commented on issue #49: Support cache for SchemaRegistryClient
Posted by "joeCarf (via GitHub)" <gi...@apache.org>.
joeCarf commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1426678615
> > 3. 由于客户端中的getschema方法需要的返回值都是GetSchemaResponse类型，因此我的缓存value也保存这个类型的值。这样的优点在于能够直接返回缓存中的value，但是缺点在于占用更多的空间，并且在register的过程中可能无法更新缓存（因为数据无法拼接成一个GetSchemaResponse）。我暂时没有想到更好的办法
>
> 最佳做法应该是在 restservice 中对 response 进行一次拆包和封装，转为类似 SchemaInfo 的类 @humkum 可以看看做一下封装
嗯嗯。我觉得或许不应该直接暴露GetSchemaResponse给用户，这样也方便在客户端进行缓存。如果 @humkum 同学在做这个封装的话，我希望能够一起合作～
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] MatrixHB commented on issue #49: Support cache for SchemaRegistryClient
Posted by GitBox <gi...@apache.org>.
MatrixHB commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1270310911
@humkum Please pay attention to [this line](https://github.com/apache/rocketmq-schema-registry/blob/70435683fac84fac0335efa646b3e94c623951fa/client/src/main/java/org/apache/rocketmq/schema/registry/client/serde/avro/AvroSerializer.java#L77), I think it will affect the performance of sending message, so client cache is very important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] humkum commented on issue #49: Support cache for SchemaRegistryClient
Posted by GitBox <gi...@apache.org>.
humkum commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1231278579
I'd like to try this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] joeCarf commented on issue #49: Support cache for SchemaRegistryClient
Posted by "joeCarf (via GitHub)" <gi...@apache.org>.
joeCarf commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1426678438
>
嗯嗯。我觉得或许不应该直接暴露GetSchemaResponse给用户，这样也方便在客户端进行缓存。如果 @humkum 同学在做这个封装的话，我希望能够一起合作～
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] joeCarf commented on issue #49: Support cache for SchemaRegistryClient
Posted by GitBox <gi...@apache.org>.
joeCarf commented on issue #49:
URL: https://github.com/apache/rocketmq-schema-registry/issues/49#issuecomment-1367910298
hi, plz assign this to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-schema-registry] ni-ze closed issue #49: Support cache for SchemaRegistryClient
Posted by "ni-ze (via GitHub)" <gi...@apache.org>.
ni-ze closed issue #49: Support cache for SchemaRegistryClient
URL: https://github.com/apache/rocketmq-schema-registry/issues/49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org