You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@shardingsphere.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2022/10/04 10:29:17 UTC

Sharingsphere for Django

Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
am considering adopting shardingshpere for the order database that requires
to cover heavy read and write. Our system is consists of django and sharded
mysql cluster with division method at this moment. I’d like to know whether
there is a reliable django library that can be used for real production.

Thanks.

-- 
Best Regards, Edward J. Yoon

Re: Sharingsphere for Django

Posted by Trista Pan <pa...@apache.org>.

Hello,

> Could you tell me about internals or why jdbc shows more good performance?

The deployment architecture of jdbc and proxy can be found in [1]. In general, jdbc is a driver embedded in your APP (similar to the mysql driver), so the request is directly from the app to the database cluster.
While proxy is deployed as a Sidecar on a pod or on a server, each request from the APP is forwarded through proxy to your database cluster (Storage cluster of ShardingSphere distributed DB system).
This case will result in additional TPC overhead (Same as other distributed DB system).

> Just found this: https://github.com/apache/shardingsphere/pull/21283

SQL hint is another useful feature for some user arbitrary visiting cases.

[1] https://shardingsphere.apache.org/document/current/en/overview/
------------------------------------------------------------------------------
Trista Pan
Twitter & Github: tristaZero

On 10/5/2022 08:10，Edward J. Yoon<ed...@apache.org> wrote：
Just found this: https://github.com/apache/shardingsphere/pull/21283

On Tue, Oct 4, 2022 at 8:35 PM Edward J. Yoon <ed...@apache.org> wrote:

Thanks Juan.

I saw some blog posts that said shardingsphere is about a 20~30%
performance penalty compared with direct connections. Could you tell me
about internals or why jdbc shows more good performance?

On Tue, Oct 4, 2022 at 8:16 PM Juan Pan <pa...@apache.org> wrote:

Jdbc is 30-40% higher than proxy based on different scenarios [1].
However, since `proxy + mysql` is similar with other distributed database
architecture,
that means proxy has the similar performance with other sharding
databases and is ready for production env (from the feedback of our
community [2, 3]).

------------------------------------------------------------------------------
Trista Pan
Twitter & Github: tristaZero

On 10/4/2022 19:05，Edward J. Yoon<ed...@apache.org> wrote：
Just curious about whether there's some performance difference between
jdbc
vs. proxy.

On Tue, Oct 4, 2022 at 8:01 PM Edward J. Yoon <ed...@apache.org>
wrote:

Currently there are 4 set consists of single master and single slave, and
one integrated db based on multi source replication. % 4 modular sharding
is used at application level. So, I'm looking for the elastic scalability,
easy-to-use/maintenance, and the high performance middleware like
shardingsphere.

On Tue, Oct 4, 2022 at 7:50 PM Juan Pan <pa...@apache.org> wrote:

Hello Edward J. Yoon,

adopting shardingshpere for the order database that requires to cover
heavy read and write.

Does that mean you want ShardingSphere to do the read-write-splitting
function?

sharded mysql cluster with division method

You already sharded your mysql database cluster? Do you plan to let
ShardingSphere help you manage all the mysql shards?

ShardingSphere currently offers two clients: ShardingSphere-JDBC (for
Java APP) and ShardingSphere-Proxy.
In your case, I recommend deploying ShardingSphere-Proxy[1] to help you
with read-write-splitting or sharding management.
Then, for your Python workload, simply connect to ShardingSphere-Proxy
using the MySQLdb library.

Best wishes,
Trista

[1]

https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/

------------------------------------------------------------------------------
Trista Pan
Twitter & Github: tristaZero

On 10/4/2022 18:29，Edward J. Yoon<ed...@apache.org> wrote：
Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
am considering adopting shardingshpere for the order database that
requires
to cover heavy read and write. Our system is consists of django and
sharded
mysql cluster with division method at this moment. I’d like to know
whether
there is a reliable django library that can be used for real production.

Thanks.

--
Best Regards, Edward J. Yoon

Re: Sharingsphere for Django

Posted by "Edward J. Yoon" <ed...@apache.org>.

Just found this: https://github.com/apache/shardingsphere/pull/21283

On Tue, Oct 4, 2022 at 8:35 PM Edward J. Yoon <ed...@apache.org> wrote:

> Thanks Juan.
>
> I saw some blog posts that said shardingsphere is about a 20~30%
> performance penalty compared with direct connections. Could you tell me
> about internals or why jdbc shows more good performance?
>
> On Tue, Oct 4, 2022 at 8:16 PM Juan Pan <pa...@apache.org> wrote:
>
>> Jdbc is 30-40% higher than proxy based on different scenarios [1].
>> However, since `proxy + mysql` is similar with other distributed database
>> architecture,
>> that means proxy has the similar performance with other sharding
>> databases and is ready for production env (from the feedback of our
>> community [2, 3]).
>>
>>
>> [1]
>> https://shardingsphere.apache.org/document/current/en/test-manual/performance-test/benchmarksql-proxy-sharding-test/
>> [2] https://shardingsphere.apache.org/blog/en/material/
>> [3] https://www.mysql.com/customers/view/?id=1461
>>
>> ------------------------------------------------------------------------------
>> Trista Pan
>> Twitter & Github: tristaZero
>>
>>
>>
>>
>> On 10/4/2022 19:05，Edward J. Yoon<ed...@apache.org> wrote：
>> Just curious about whether there's some performance difference between
>> jdbc
>> vs. proxy.
>>
>> On Tue, Oct 4, 2022 at 8:01 PM Edward J. Yoon <ed...@apache.org>
>> wrote:
>>
>> Currently there are 4 set consists of single master and single slave, and
>> one integrated db based on multi source replication. % 4 modular sharding
>> is used at application level. So, I'm looking for the elastic scalability,
>> easy-to-use/maintenance, and the high performance middleware like
>> shardingsphere.
>>
>> On Tue, Oct 4, 2022 at 7:50 PM Juan Pan <pa...@apache.org> wrote:
>>
>> Hello Edward J. Yoon,
>>
>>
>> adopting shardingshpere for the order database that requires to cover
>> heavy read and write.
>>
>>
>> Does that mean you want ShardingSphere to do the read-write-splitting
>> function?
>>
>>
>> sharded mysql cluster with division method
>>
>>
>> You already sharded your mysql database cluster? Do you plan to let
>> ShardingSphere help you manage all the mysql shards?
>>
>>
>> ShardingSphere currently offers two clients: ShardingSphere-JDBC (for
>> Java APP) and ShardingSphere-Proxy.
>> In your case, I recommend deploying ShardingSphere-Proxy[1] to help you
>> with read-write-splitting or sharding management.
>> Then, for your Python workload, simply connect to ShardingSphere-Proxy
>> using the MySQLdb library.
>>
>> Best wishes,
>> Trista
>>
>>
>> [1]
>>
>> https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/
>>
>>
>> ------------------------------------------------------------------------------
>> Trista Pan
>> Twitter & Github: tristaZero
>>
>>
>>
>>
>> On 10/4/2022 18:29，Edward J. Yoon<ed...@apache.org> wrote：
>> Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
>> am considering adopting shardingshpere for the order database that
>> requires
>> to cover heavy read and write. Our system is consists of django and
>> sharded
>> mysql cluster with division method at this moment. I’d like to know
>> whether
>> there is a reliable django library that can be used for real production.
>>
>> Thanks.
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>
>
> --
> Best Regards, Edward J. Yoon
>


-- 
Best Regards, Edward J. Yoon

Re: Sharingsphere for Django

Posted by "Edward J. Yoon" <ed...@apache.org>.

Thanks Juan.

I saw some blog posts that said shardingsphere is about a 20~30%
performance penalty compared with direct connections. Could you tell me
about internals or why jdbc shows more good performance?

On Tue, Oct 4, 2022 at 8:16 PM Juan Pan <pa...@apache.org> wrote:

> Jdbc is 30-40% higher than proxy based on different scenarios [1].
> However, since `proxy + mysql` is similar with other distributed database
> architecture,
> that means proxy has the similar performance with other sharding databases
> and is ready for production env (from the feedback of our community [2, 3]).
>
>
> [1]
> https://shardingsphere.apache.org/document/current/en/test-manual/performance-test/benchmarksql-proxy-sharding-test/
> [2] https://shardingsphere.apache.org/blog/en/material/
> [3] https://www.mysql.com/customers/view/?id=1461
>
> ------------------------------------------------------------------------------
> Trista Pan
> Twitter & Github: tristaZero
>
>
>
>
> On 10/4/2022 19:05，Edward J. Yoon<ed...@apache.org> wrote：
> Just curious about whether there's some performance difference between jdbc
> vs. proxy.
>
> On Tue, Oct 4, 2022 at 8:01 PM Edward J. Yoon <ed...@apache.org>
> wrote:
>
> Currently there are 4 set consists of single master and single slave, and
> one integrated db based on multi source replication. % 4 modular sharding
> is used at application level. So, I'm looking for the elastic scalability,
> easy-to-use/maintenance, and the high performance middleware like
> shardingsphere.
>
> On Tue, Oct 4, 2022 at 7:50 PM Juan Pan <pa...@apache.org> wrote:
>
> Hello Edward J. Yoon,
>
>
> adopting shardingshpere for the order database that requires to cover
> heavy read and write.
>
>
> Does that mean you want ShardingSphere to do the read-write-splitting
> function?
>
>
> sharded mysql cluster with division method
>
>
> You already sharded your mysql database cluster? Do you plan to let
> ShardingSphere help you manage all the mysql shards?
>
>
> ShardingSphere currently offers two clients: ShardingSphere-JDBC (for
> Java APP) and ShardingSphere-Proxy.
> In your case, I recommend deploying ShardingSphere-Proxy[1] to help you
> with read-write-splitting or sharding management.
> Then, for your Python workload, simply connect to ShardingSphere-Proxy
> using the MySQLdb library.
>
> Best wishes,
> Trista
>
>
> [1]
>
> https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/
>
>
> ------------------------------------------------------------------------------
> Trista Pan
> Twitter & Github: tristaZero
>
>
>
>
> On 10/4/2022 18:29，Edward J. Yoon<ed...@apache.org> wrote：
> Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
> am considering adopting shardingshpere for the order database that
> requires
> to cover heavy read and write. Our system is consists of django and
> sharded
> mysql cluster with division method at this moment. I’d like to know
> whether
> there is a reliable django library that can be used for real production.
>
> Thanks.
>
> --
> Best Regards, Edward J. Yoon
>
>
>
> --
> Best Regards, Edward J. Yoon
>
>
>
> --
> Best Regards, Edward J. Yoon
>


-- 
Best Regards, Edward J. Yoon

Re: Sharingsphere for Django

Posted by Juan Pan <pa...@apache.org>.

Jdbc is 30-40% higher than proxy based on different scenarios [1].
However, since `proxy + mysql` is similar with other distributed database architecture,
that means proxy has the similar performance with other sharding databases and is ready for production env (from the feedback of our community [2, 3]).

[1] https://shardingsphere.apache.org/document/current/en/test-manual/performance-test/benchmarksql-proxy-sharding-test/
[2] https://shardingsphere.apache.org/blog/en/material/
[3] https://www.mysql.com/customers/view/?id=1461
------------------------------------------------------------------------------
Trista Pan
Twitter & Github: tristaZero

On 10/4/2022 19:05，Edward J. Yoon<ed...@apache.org> wrote：
Just curious about whether there's some performance difference between jdbc
vs. proxy.

On Tue, Oct 4, 2022 at 8:01 PM Edward J. Yoon <ed...@apache.org> wrote:

On Tue, Oct 4, 2022 at 7:50 PM Juan Pan <pa...@apache.org> wrote:

Hello Edward J. Yoon,

adopting shardingshpere for the order database that requires to cover
heavy read and write.

Does that mean you want ShardingSphere to do the read-write-splitting
function?

sharded mysql cluster with division method

You already sharded your mysql database cluster? Do you plan to let
ShardingSphere help you manage all the mysql shards?

Best wishes,
Trista

[1]
https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/

------------------------------------------------------------------------------
Trista Pan
Twitter & Github: tristaZero

Thanks.

--
Best Regards, Edward J. Yoon

Re: Sharingsphere for Django

Posted by "Edward J. Yoon" <ed...@apache.org>.

Just curious about whether there's some performance difference between jdbc
vs. proxy.

On Tue, Oct 4, 2022 at 8:01 PM Edward J. Yoon <ed...@apache.org> wrote:

> Currently there are 4 set consists of single master and single slave, and
> one integrated db based on multi source replication. % 4 modular sharding
> is used at application level. So, I'm looking for the elastic scalability,
> easy-to-use/maintenance, and the high performance middleware like
> shardingsphere.
>
> On Tue, Oct 4, 2022 at 7:50 PM Juan Pan <pa...@apache.org> wrote:
>
>> Hello Edward J. Yoon,
>>
>>
>> > adopting shardingshpere for the order database that requires to cover
>> heavy read and write.
>>
>>
>> Does that mean you want ShardingSphere to do the read-write-splitting
>> function?
>>
>>
>> > sharded mysql cluster with division method
>>
>>
>> You already sharded your mysql database cluster? Do you plan to let
>> ShardingSphere help you manage all the mysql shards?
>>
>>
>> ShardingSphere currently offers two clients: ShardingSphere-JDBC (for
>> Java APP) and ShardingSphere-Proxy.
>> In your case, I recommend deploying ShardingSphere-Proxy[1] to help you
>> with read-write-splitting or sharding management.
>> Then, for your Python workload, simply connect to ShardingSphere-Proxy
>> using the MySQLdb library.
>>
>> Best wishes,
>> Trista
>>
>>
>> [1]
>> https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/
>>
>> ------------------------------------------------------------------------------
>> Trista Pan
>> Twitter & Github: tristaZero
>>
>>
>>
>>
>> On 10/4/2022 18:29，Edward J. Yoon<ed...@apache.org> wrote：
>> Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
>> am considering adopting shardingshpere for the order database that
>> requires
>> to cover heavy read and write. Our system is consists of django and
>> sharded
>> mysql cluster with division method at this moment. I’d like to know
>> whether
>> there is a reliable django library that can be used for real production.
>>
>> Thanks.
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>
>
> --
> Best Regards, Edward J. Yoon
>


-- 
Best Regards, Edward J. Yoon

Re: Sharingsphere for Django

Posted by "Edward J. Yoon" <ed...@apache.org>.

Currently there are 4 set consists of single master and single slave, and
one integrated db based on multi source replication. % 4 modular sharding
is used at application level. So, I'm looking for the elastic scalability,
easy-to-use/maintenance, and the high performance middleware like
shardingsphere.

On Tue, Oct 4, 2022 at 7:50 PM Juan Pan <pa...@apache.org> wrote:

> Hello Edward J. Yoon,
>
>
> > adopting shardingshpere for the order database that requires to cover
> heavy read and write.
>
>
> Does that mean you want ShardingSphere to do the read-write-splitting
> function?
>
>
> > sharded mysql cluster with division method
>
>
> You already sharded your mysql database cluster? Do you plan to let
> ShardingSphere help you manage all the mysql shards?
>
>
> ShardingSphere currently offers two clients: ShardingSphere-JDBC (for Java
> APP) and ShardingSphere-Proxy.
> In your case, I recommend deploying ShardingSphere-Proxy[1] to help you
> with read-write-splitting or sharding management.
> Then, for your Python workload, simply connect to ShardingSphere-Proxy
> using the MySQLdb library.
>
> Best wishes,
> Trista
>
>
> [1]
> https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/
>
> ------------------------------------------------------------------------------
> Trista Pan
> Twitter & Github: tristaZero
>
>
>
>
> On 10/4/2022 18:29，Edward J. Yoon<ed...@apache.org> wrote：
> Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
> am considering adopting shardingshpere for the order database that requires
> to cover heavy read and write. Our system is consists of django and sharded
> mysql cluster with division method at this moment. I’d like to know whether
> there is a reliable django library that can be used for real production.
>
> Thanks.
>
> --
> Best Regards, Edward J. Yoon
>


-- 
Best Regards, Edward J. Yoon

Re:Sharingsphere for Django

Posted by Juan Pan <pa...@apache.org>.

Hello Edward J. Yoon,


> adopting shardingshpere for the order database that requires to cover heavy read and write. 


Does that mean you want ShardingSphere to do the read-write-splitting function?


> sharded mysql cluster with division method 


You already sharded your mysql database cluster? Do you plan to let ShardingSphere help you manage all the mysql shards?


ShardingSphere currently offers two clients: ShardingSphere-JDBC (for Java APP) and ShardingSphere-Proxy.
In your case, I recommend deploying ShardingSphere-Proxy[1] to help you with read-write-splitting or sharding management.
Then, for your Python workload, simply connect to ShardingSphere-Proxy using the MySQLdb library.
 
Best wishes,
Trista


[1] https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/
------------------------------------------------------------------------------
Trista Pan
Twitter & Github: tristaZero




On 10/4/2022 18:29，Edward J. Yoon<ed...@apache.org> wrote：
Hello, I am edwardyoon, currently working at Delivery Hero. From there, I
am considering adopting shardingshpere for the order database that requires
to cover heavy read and write. Our system is consists of django and sharded
mysql cluster with division method at this moment. I’d like to know whether
there is a reliable django library that can be used for real production.

Thanks.

--
Best Regards, Edward J. Yoon