回复: Re:Re: some suggesions for sharding scaling

[Sun Nianjun <ka...@outlook.com>]

thank you fellows, I do want to commit something useful to community .
________________________________
发件人: kimmking@163.com <ki...@163.com> 代表 KimmKing <ki...@apache.org>
发送时间: 2020年6月15日 16:20
收件人: dev@shardingsphere.apache.org <de...@shardingsphere.apache.org>
主题: Re:Re: some suggesions for sharding scaling

Hi, Nianjun.
BTW, about the single proxy bottleneck, we will try to resolve it with proxies cluster in control-panel module, you can track and join it.
>5. ss-proxy is the first version, we will plug ss-scaling into ss-jdbc in the future, so don't warry about the proxy bottleneck.





At 2020-06-15 15:53:15, "lucas" <lu...@163.com> wrote:
>Hi, Sun Nianjun:
>
>
>Think you for your suggesions.
>
>
>1. Your understanding is pretty good. Change `offer` to `put` is a good idea, you can do it. And I think queue size defalut as 10000 is ok, if it's too big, I'm a litter warry memory. as you know, it is multithreading task andconsumer can change it by configuration.
>
>
>2. Breakpoint resume is a very important function, and there is other guy do it now.
>
>
>3. You are right, we are think about to use elastic-job to process scaling job.
>
>
>4. You suggesion is very good, but we want all put all other functions (such as encryption) into ss-jdbc, and scaling is only scaling.
>
>
>5. ss-proxy is the first version, we will plug ss-scaling into ss-jdbc in the future, so don't warry about the proxy bottleneck.
>
>
>6. `/scaling/job/start` supports json formats, see: org.apache.shardingsphere.shardingscaling.web.HttpServerHandler#startJob
>
>
>You suggesions are very visionary. there are lot of things we should do it in the future. and welcome to do it together.
>
>
>
>
>-------- Forwarding messages --------
>
>From: "Sun Nianjun" <ka...@outlook.com>
>Date: 2020-06-14 23:54:38
>To:  "dev@shardingsphere.apache.org" <de...@shardingsphere.apache.org>
>Subject: some suggesions for sharding scaling
>hi , commuity
>
>    since last week, I got some experience of using sharding scaling, and  I have summarized some suggestions as follows, hope to discuss with the community :
>
>
>  1.   dumper, importer, channel is a typical producer-consumer pattern， mainly used BlockingQueue in Java，and I adjusted the details of the use a little in my actual experience: I changed the 'offer' to 'put' in MemoryChannel , change the queue size a little bit larger, because the dumper will stuffed the queue full very fast, and dumper is relative slow, even there is a timeout limitation for method `offer`, it's very easy to cause an RuntimeException by a failed offer.
>  2.  I’m thinking about how to add a breakpoint resume function to scaling, because once there is a problem during the migration process, it is very troublesome。maybe we could record the scaling position through local log or something like that, we have to make sure the scaling could resume through breakpoint after broken down.
>  3.  since the elastic job is resurrected, why not seperate and handle the job in scaling by elastic job ? a job system must be more Robust and Stable
>  4.  It's very common usage in shardingsphere to load an implementaion for interface, that's SPI . I think the current dumper and importer should exact as default implementation, and add more implementions and load by SPI . BTW , importer is not just focus on import data into database, maybe for encryption.
>  5.  I believe proxy will be the bottleneck for current architecture , event I tried to scaling more than 40 million data in a mysql to 8 new shard , sharding proxy is a little bit 'slow' , if we prepare to scale billions data from 16 sharding to 32 shard , I don't think proxy could withstand such pressure，we may use sharding jdbc for scaling to reshard.
>  6.  currently, the parameters for `/scaling/job/start` are in yaml formats, it's ok if that's for API invocation , but its not friendly for debugging，should we add some function to translate yaml configuration to json format in API invocation ？
>
>shardingsphere has a very active and open community ,  I hope the metors could give some suggestions for that .
>
>Regards
>
>
>
>
>
>