You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apisix.apache.org by Baoyuan <ba...@gmail.com> on 2022/04/02 00:52:59 UTC

Re: Proposal: reduce connections between Manager API and etcd

Hi, I agree with that.

Chao Zhang <to...@apache.org> 于2022年4月2日周六 07:56写道：

> Hi,
>
> SGTM on my side.
>
> JunXu Chen <ch...@apache.org>于2022年3月29日 周二10:48写道：
>
> > # Background
> > At present, the connection between the Manager API and etcd still follows
> > the habit of the etcd v2 version, creating a watch connection for each
> > resource, which will generate a large number of connections.
> >
> > # Purpose
> > Reduce connections between Manager API and etcd, to reduce the load on
> etcd
> > services.
> >
> > # Proposal
> > In the `Generic Store`, each resource still manages its own cache
> > separately, but no longer connects directly to etcd. Added a unified
> method
> > of listing full data and watching changes, and the obtained data is
> > uniformly distributed to the Store instances of each resource.
> > After optimization, the workflow of `Generic Store` initialization is as
> > follows:
> > 1. Create a resource prefix mapping table
> > 2. Initialize each resource Store object and register itself in the
> > resource prefix mapping table
> > 3. Read the full amount of data, find the corresponding resource
> according
> > to the key prefix, and distribute the data to the Store object of each
> > resource
> > 4. Watch the etcd prefix configured in `conf.yaml`, when obtaining the
> > changed data, find the corresponding resource according to the key
> prefix,
> > and distribute the data to the Store object of the resource.
> >
> > # Further action
> > Reduce connections between Apache APISIX and etcd (a new proposal is
> > needed)
> >
> > I also drew a flow chart of the comparison before and after optimization
> in
> > the issue. Welcome to view it and discuss in the issue[1].
> >
> > Thanks!
> >
> >
> > [1] https://github.com/apache/apisix-dashboard/issues/2396
> >
> --
> Best regards
> Chao Zhang
>
> https://github.com/tokers
>

Re: Proposal: reduce connections between Manager API and etcd

Posted by wei jin <kv...@apache.org>.

LGTM, Avoid etcd cluster with too many connections.

Qi Guo <gu...@gmail.com> 于2022年4月2日周六 10:08写道：

> Agree +1
>
> Leslie Tsang <le...@apache.org> 于2022年4月2日周六 09:54写道：
>
> > SGTM, This means that the etcd cluster can support more APISIX nodes than
> > ever before.
> >
> > Leslie Tsang
> > leslie@apache.org
> >
> > > On 2 Apr 2022, at 9:51 AM, Wei Jiang <ma...@gmail.com> wrote:
> > >
> > > LGTM, Agree +1
> > >
> > >> 2022年4月2日 上午8:52，Baoyuan <ba...@gmail.com> 写道：
> > >>
> > >> Hi, I agree with that.
> > >>
> > >> Chao Zhang <to...@apache.org> 于2022年4月2日周六 07:56写道：
> > >>
> > >>> Hi,
> > >>>
> > >>> SGTM on my side.
> > >>>
> > >>> JunXu Chen <ch...@apache.org>于2022年3月29日 周二10:48写道：
> > >>>
> > >>>> # Background
> > >>>> At present, the connection between the Manager API and etcd still
> > follows
> > >>>> the habit of the etcd v2 version, creating a watch connection for
> each
> > >>>> resource, which will generate a large number of connections.
> > >>>>
> > >>>> # Purpose
> > >>>> Reduce connections between Manager API and etcd, to reduce the load
> on
> > >>> etcd
> > >>>> services.
> > >>>>
> > >>>> # Proposal
> > >>>> In the `Generic Store`, each resource still manages its own cache
> > >>>> separately, but no longer connects directly to etcd. Added a unified
> > >>> method
> > >>>> of listing full data and watching changes, and the obtained data is
> > >>>> uniformly distributed to the Store instances of each resource.
> > >>>> After optimization, the workflow of `Generic Store` initialization
> is
> > as
> > >>>> follows:
> > >>>> 1. Create a resource prefix mapping table
> > >>>> 2. Initialize each resource Store object and register itself in the
> > >>>> resource prefix mapping table
> > >>>> 3. Read the full amount of data, find the corresponding resource
> > >>> according
> > >>>> to the key prefix, and distribute the data to the Store object of
> each
> > >>>> resource
> > >>>> 4. Watch the etcd prefix configured in `conf.yaml`, when obtaining
> the
> > >>>> changed data, find the corresponding resource according to the key
> > >>> prefix,
> > >>>> and distribute the data to the Store object of the resource.
> > >>>>
> > >>>> # Further action
> > >>>> Reduce connections between Apache APISIX and etcd (a new proposal is
> > >>>> needed)
> > >>>>
> > >>>> I also drew a flow chart of the comparison before and after
> > optimization
> > >>> in
> > >>>> the issue. Welcome to view it and discuss in the issue[1].
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>>
> > >>>> [1] https://github.com/apache/apisix-dashboard/issues/2396
> > >>>>
> > >>> --
> > >>> Best regards
> > >>> Chao Zhang
> > >>>
> > >>> https://github.com/tokers
> > >>>
> > >
> >
> >
>

Re: Proposal: reduce connections between Manager API and etcd

Posted by Qi Guo <gu...@gmail.com>.

Agree +1

Leslie Tsang <le...@apache.org> 于2022年4月2日周六 09:54写道：

> SGTM, This means that the etcd cluster can support more APISIX nodes than
> ever before.
>
> Leslie Tsang
> leslie@apache.org
>
> > On 2 Apr 2022, at 9:51 AM, Wei Jiang <ma...@gmail.com> wrote:
> >
> > LGTM, Agree +1
> >
> >> 2022年4月2日 上午8:52，Baoyuan <ba...@gmail.com> 写道：
> >>
> >> Hi, I agree with that.
> >>
> >> Chao Zhang <to...@apache.org> 于2022年4月2日周六 07:56写道：
> >>
> >>> Hi,
> >>>
> >>> SGTM on my side.
> >>>
> >>> JunXu Chen <ch...@apache.org>于2022年3月29日 周二10:48写道：
> >>>
> >>>> # Background
> >>>> At present, the connection between the Manager API and etcd still
> follows
> >>>> the habit of the etcd v2 version, creating a watch connection for each
> >>>> resource, which will generate a large number of connections.
> >>>>
> >>>> # Purpose
> >>>> Reduce connections between Manager API and etcd, to reduce the load on
> >>> etcd
> >>>> services.
> >>>>
> >>>> # Proposal
> >>>> In the `Generic Store`, each resource still manages its own cache
> >>>> separately, but no longer connects directly to etcd. Added a unified
> >>> method
> >>>> of listing full data and watching changes, and the obtained data is
> >>>> uniformly distributed to the Store instances of each resource.
> >>>> After optimization, the workflow of `Generic Store` initialization is
> as
> >>>> follows:
> >>>> 1. Create a resource prefix mapping table
> >>>> 2. Initialize each resource Store object and register itself in the
> >>>> resource prefix mapping table
> >>>> 3. Read the full amount of data, find the corresponding resource
> >>> according
> >>>> to the key prefix, and distribute the data to the Store object of each
> >>>> resource
> >>>> 4. Watch the etcd prefix configured in `conf.yaml`, when obtaining the
> >>>> changed data, find the corresponding resource according to the key
> >>> prefix,
> >>>> and distribute the data to the Store object of the resource.
> >>>>
> >>>> # Further action
> >>>> Reduce connections between Apache APISIX and etcd (a new proposal is
> >>>> needed)
> >>>>
> >>>> I also drew a flow chart of the comparison before and after
> optimization
> >>> in
> >>>> the issue. Welcome to view it and discuss in the issue[1].
> >>>>
> >>>> Thanks!
> >>>>
> >>>>
> >>>> [1] https://github.com/apache/apisix-dashboard/issues/2396
> >>>>
> >>> --
> >>> Best regards
> >>> Chao Zhang
> >>>
> >>> https://github.com/tokers
> >>>
> >
>
>

Re: Proposal: reduce connections between Manager API and etcd

Posted by Leslie Tsang <le...@apache.org>.

SGTM, This means that the etcd cluster can support more APISIX nodes than ever before.

Leslie Tsang
leslie@apache.org

> On 2 Apr 2022, at 9:51 AM, Wei Jiang <ma...@gmail.com> wrote:
> 
> LGTM, Agree +1
> 
>> 2022年4月2日 上午8:52，Baoyuan <ba...@gmail.com> 写道：
>> 
>> Hi, I agree with that.
>> 
>> Chao Zhang <to...@apache.org> 于2022年4月2日周六 07:56写道：
>> 
>>> Hi,
>>> 
>>> SGTM on my side.
>>> 
>>> JunXu Chen <ch...@apache.org>于2022年3月29日 周二10:48写道：
>>> 
>>>> # Background
>>>> At present, the connection between the Manager API and etcd still follows
>>>> the habit of the etcd v2 version, creating a watch connection for each
>>>> resource, which will generate a large number of connections.
>>>> 
>>>> # Purpose
>>>> Reduce connections between Manager API and etcd, to reduce the load on
>>> etcd
>>>> services.
>>>> 
>>>> # Proposal
>>>> In the `Generic Store`, each resource still manages its own cache
>>>> separately, but no longer connects directly to etcd. Added a unified
>>> method
>>>> of listing full data and watching changes, and the obtained data is
>>>> uniformly distributed to the Store instances of each resource.
>>>> After optimization, the workflow of `Generic Store` initialization is as
>>>> follows:
>>>> 1. Create a resource prefix mapping table
>>>> 2. Initialize each resource Store object and register itself in the
>>>> resource prefix mapping table
>>>> 3. Read the full amount of data, find the corresponding resource
>>> according
>>>> to the key prefix, and distribute the data to the Store object of each
>>>> resource
>>>> 4. Watch the etcd prefix configured in `conf.yaml`, when obtaining the
>>>> changed data, find the corresponding resource according to the key
>>> prefix,
>>>> and distribute the data to the Store object of the resource.
>>>> 
>>>> # Further action
>>>> Reduce connections between Apache APISIX and etcd (a new proposal is
>>>> needed)
>>>> 
>>>> I also drew a flow chart of the comparison before and after optimization
>>> in
>>>> the issue. Welcome to view it and discuss in the issue[1].
>>>> 
>>>> Thanks!
>>>> 
>>>> 
>>>> [1] https://github.com/apache/apisix-dashboard/issues/2396
>>>> 
>>> --
>>> Best regards
>>> Chao Zhang
>>> 
>>> https://github.com/tokers
>>> 
>

Re: Proposal: reduce connections between Manager API and etcd

Posted by Wei Jiang <ma...@gmail.com>.

LGTM, Agree +1

> 2022年4月2日 上午8:52，Baoyuan <ba...@gmail.com> 写道：
> 
> Hi, I agree with that.
> 
> Chao Zhang <to...@apache.org> 于2022年4月2日周六 07:56写道：
> 
>> Hi,
>> 
>> SGTM on my side.
>> 
>> JunXu Chen <ch...@apache.org>于2022年3月29日 周二10:48写道：
>> 
>>> # Background
>>> At present, the connection between the Manager API and etcd still follows
>>> the habit of the etcd v2 version, creating a watch connection for each
>>> resource, which will generate a large number of connections.
>>> 
>>> # Purpose
>>> Reduce connections between Manager API and etcd, to reduce the load on
>> etcd
>>> services.
>>> 
>>> # Proposal
>>> In the `Generic Store`, each resource still manages its own cache
>>> separately, but no longer connects directly to etcd. Added a unified
>> method
>>> of listing full data and watching changes, and the obtained data is
>>> uniformly distributed to the Store instances of each resource.
>>> After optimization, the workflow of `Generic Store` initialization is as
>>> follows:
>>> 1. Create a resource prefix mapping table
>>> 2. Initialize each resource Store object and register itself in the
>>> resource prefix mapping table
>>> 3. Read the full amount of data, find the corresponding resource
>> according
>>> to the key prefix, and distribute the data to the Store object of each
>>> resource
>>> 4. Watch the etcd prefix configured in `conf.yaml`, when obtaining the
>>> changed data, find the corresponding resource according to the key
>> prefix,
>>> and distribute the data to the Store object of the resource.
>>> 
>>> # Further action
>>> Reduce connections between Apache APISIX and etcd (a new proposal is
>>> needed)
>>> 
>>> I also drew a flow chart of the comparison before and after optimization
>> in
>>> the issue. Welcome to view it and discuss in the issue[1].
>>> 
>>> Thanks!
>>> 
>>> 
>>> [1] https://github.com/apache/apisix-dashboard/issues/2396
>>> 
>> --
>> Best regards
>> Chao Zhang
>> 
>> https://github.com/tokers
>>

Re: Proposal: reduce connections between Manager API and etcd

Posted by Bozhong Yu <im...@gmail.com>.

Agree +1

Baoyuan <ba...@gmail.com> 于2022年4月2日周六 08:53写道：

> Hi, I agree with that.
>
> Chao Zhang <to...@apache.org> 于2022年4月2日周六 07:56写道：
>
> > Hi,
> >
> > SGTM on my side.
> >
> > JunXu Chen <ch...@apache.org>于2022年3月29日 周二10:48写道：
> >
> > > # Background
> > > At present, the connection between the Manager API and etcd still
> follows
> > > the habit of the etcd v2 version, creating a watch connection for each
> > > resource, which will generate a large number of connections.
> > >
> > > # Purpose
> > > Reduce connections between Manager API and etcd, to reduce the load on
> > etcd
> > > services.
> > >
> > > # Proposal
> > > In the `Generic Store`, each resource still manages its own cache
> > > separately, but no longer connects directly to etcd. Added a unified
> > method
> > > of listing full data and watching changes, and the obtained data is
> > > uniformly distributed to the Store instances of each resource.
> > > After optimization, the workflow of `Generic Store` initialization is
> as
> > > follows:
> > > 1. Create a resource prefix mapping table
> > > 2. Initialize each resource Store object and register itself in the
> > > resource prefix mapping table
> > > 3. Read the full amount of data, find the corresponding resource
> > according
> > > to the key prefix, and distribute the data to the Store object of each
> > > resource
> > > 4. Watch the etcd prefix configured in `conf.yaml`, when obtaining the
> > > changed data, find the corresponding resource according to the key
> > prefix,
> > > and distribute the data to the Store object of the resource.
> > >
> > > # Further action
> > > Reduce connections between Apache APISIX and etcd (a new proposal is
> > > needed)
> > >
> > > I also drew a flow chart of the comparison before and after
> optimization
> > in
> > > the issue. Welcome to view it and discuss in the issue[1].
> > >
> > > Thanks!
> > >
> > >
> > > [1] https://github.com/apache/apisix-dashboard/issues/2396
> > >
> > --
> > Best regards
> > Chao Zhang
> >
> > https://github.com/tokers
> >
>