You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2021/12/15 16:24:59 UTC

[GitHub] [servicecomb-service-center] little-cui opened a new pull request #1180: Feature: Add schema ref

little-cui opened a new pull request #1180:
URL: https://github.com/apache/servicecomb-service-center/pull/1180


   ### 背景
   
   当前契约是按照微服务版本一比一存储的，当升级微服务版本，较多场景下会冗余schema存储，导致后端数据存储暴涨
   
   ### 实现方式
   
   - hash：使用md5算法hash文件名与内容（要做接口抽象，以便后续切换算法），算出每个契约文件的“地址”，将地址作为etcd key，内容作为etcd value存储，不要做全量缓存，会把内存挤爆的，为了优化查询速度，只能做热点查询，比如就留1000size的契约缓存。
   
   - 缓存：可以缓存{serviceid}/{schemaid}到hash(schemaid+schemacontent)的映射进行加速
   
   - 存储结构：/cse-sr/ms/schema-content/{domain}/{project}/{schema_hash}，值为契约名字和内容
   
   - 汇总查询：service的schemas slice应该存的是这些地址，需要汇总时，再进行关联查询
   
    
   ### 老数据兼容、老接口兼容的总原则
   
   - 新老数据共存
   - 不增API
   - 允许一个微服务版本部分schema是老逻辑，部分是新逻辑
   
   Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/browse/SCB) filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
    - [ ] Each commit in the pull request should have a meaningful subject line and body.
    - [ ] Format the pull request title like `[SCB-XXX] Fixes bug in ApproximateQuantiles`, where you replace `SCB-XXX` with the appropriate JIRA issue.
    - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
    - [ ] Run `go build` `go test` `go fmt` `go vet` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    - [ ] Never comment source code, delete it.
    - [ ] UT should has "context, subject, expected result" result as test case name, when you call t.Run().
   ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-service-center] robotLJW commented on a change in pull request #1180: Feature: Add schema ref

Posted by GitBox <gi...@apache.org>.
robotLJW commented on a change in pull request #1180:
URL: https://github.com/apache/servicecomb-service-center/pull/1180#discussion_r770162729



##########
File path: datasource/etcd/schema.go
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package etcd
+
+import (
+	"context"
+
+	"github.com/apache/servicecomb-service-center/datasource/schema"
+)
+
+func init() {
+	schema.Install("etcd", NewSchemaDAO)
+	schema.Install("embeded_etcd", NewSchemaDAO)
+	schema.Install("embedded_etcd", NewSchemaDAO)
+}
+
+func NewSchemaDAO(opts schema.Options) (schema.DAO, error) {
+	return &SchemaDAO{}, nil
+}
+
+type SchemaDAO struct{}
+
+func (SchemaDAO) GetRef(ctx context.Context, ref *schema.Ref) (*schema.Ref, error) {
+	panic("implement me")
+}
+
+func (SchemaDAO) PutRef(ctx context.Context, ref *schema.Ref) error {
+	panic("implement me")
+}
+
+func (SchemaDAO) DeleteRef(ctx context.Context, ref ...*schema.Ref) error {
+	panic("implement me")
+}
+
+func (SchemaDAO) GetContent(ctx context.Context, hash *schema.ContentRequest) (string, error) {

Review comment:
       hash 是不是换成 req 更合适一点




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-service-center] robotLJW commented on a change in pull request #1180: Feature: Add schema ref

Posted by GitBox <gi...@apache.org>.
robotLJW commented on a change in pull request #1180:
URL: https://github.com/apache/servicecomb-service-center/pull/1180#discussion_r770156060



##########
File path: datasource/etcd/schema.go
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package etcd
+
+import (
+	"context"
+
+	"github.com/apache/servicecomb-service-center/datasource/schema"
+)
+
+func init() {
+	schema.Install("etcd", NewSchemaDAO)
+	schema.Install("embeded_etcd", NewSchemaDAO)
+	schema.Install("embedded_etcd", NewSchemaDAO)

Review comment:
       这边有3种类型？应该只有2种吧




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-service-center] tianxiaoliang commented on pull request #1180: Feature: Add schema ref

Posted by GitBox <gi...@apache.org>.
tianxiaoliang commented on pull request #1180:
URL: https://github.com/apache/servicecomb-service-center/pull/1180#issuecomment-1078679778


   that is really a great pull request for cost down


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-service-center] tianxiaoliang merged pull request #1180: Feature: Add schema ref

Posted by GitBox <gi...@apache.org>.
tianxiaoliang merged pull request #1180:
URL: https://github.com/apache/servicecomb-service-center/pull/1180


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-service-center] tianxiaoliang commented on a change in pull request #1180: Feature: Add schema ref

Posted by GitBox <gi...@apache.org>.
tianxiaoliang commented on a change in pull request #1180:
URL: https://github.com/apache/servicecomb-service-center/pull/1180#discussion_r770157740



##########
File path: datasource/etcd/schema.go
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package etcd
+
+import (
+	"context"
+
+	"github.com/apache/servicecomb-service-center/datasource/schema"
+)
+
+func init() {
+	schema.Install("etcd", NewSchemaDAO)
+	schema.Install("embeded_etcd", NewSchemaDAO)
+	schema.Install("embedded_etcd", NewSchemaDAO)

Review comment:
       这是为了兼容拼写错误的老插件名字




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org