You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Igor Sapego <is...@apache.org> on 2022/05/16 20:07:26 UTC
IEP-90 Ignite 3 Client Lifecycle
Hi, Igniters
I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main idea is to
define client lifecycle as well as core algorithms and mechanisms used by
clients. This proposal can be used as a reference for implementation of a
new client for Ignite when dealing with such problems as:
- Resolving of user-provided addresses;
- Initial connection to a cluster;
- Maintaining cluster connection;
- Connection recovery;
- Connection break handling.
So take a look and let me know what you think guys.
[1] -
https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
Best Regards,
Igor
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
One more corner case:
In case of a connection loss, even if the message was successfully written
to the socket, there is no guarantee it was read on the other side
For example, a client could have sent a request with ID 10, but the server
did not get it.
Or the server sent a reply to request 9, but the client did not get it.
So when a logical connection is restored, we need to compare pending client
requests (TcpClientChannel.pendingReqs) with the server outbox.
To do that, I propose to send a list of known requestIds from the server to
the client.
The client then completes missing requests with an error and RetryPolicy
will deal with them up the call stack.
On Fri, May 20, 2022 at 12:02 PM Pavel Tupitsyn <pt...@apache.org>
wrote:
> Another point:
>
> With the current proposal, there is no way for the server to distinguish
> a broken connection from a normal user-initiated connection close.
>
> So in either case we'll wait for some configured timeout before cleaning
> up session resources.
> I think this is fine, at least for now.
>
> On Thu, May 19, 2022 at 10:55 PM Igor Sapego <is...@apache.org> wrote:
>
>> Andrey,
>>
>> 1. If a server generates a UUID that already exists it can check and just
>> re-generate it straight away
>> as a check is just a simple map lookup.
>>
>> 2. Well, yes. This proposal does not consider monitoring, but with current
>> approach it will be definitely
>> easier to implement it properly.
>>
>> Pavel,
>>
>> Wow, that's really cool that you've already started working on it.
>>
>> Regarding your proposal I like it - it will really make a procedure easier
>> and faster for a client. I'll change
>> the IEP accordingly.
>>
>> Best Regards,
>> Igor
>>
>>
>> On Thu, May 19, 2022 at 11:40 PM Pavel Tupitsyn <pt...@apache.org>
>> wrote:
>>
>> > Igor,
>> >
>> > I've started working on the initial sessions implementation [1],
>> > and I'd like to clarify the procedure of logical connection restore.
>> >
>> > > client in its turn tries to establish a new *node connection*
>> > > and restore *logical connection *using ConnectionRestoreReq
>> >
>> > This implies either an additional request, or something that replaces
>> > normal handshake.
>> > However, we need to handle two cases here:
>> > - Logical connection is restored
>> > - Logical connection is not restored (timed out, server restarted,
>> etc), in
>> > which case we still establish the connection and use it.
>> >
>> > To account for the second case, we should always start with a regular
>> > handshake.
>> > I propose to add a nullable UUID field to the handshake request to cover
>> > both cases:
>> > - connectionId is null or not found on server: proceed with normal
>> > handshake, return newly generated connectionId.
>> > - connectionId is not null and found on server: restore logical
>> connection,
>> > return the same connectionId
>> >
>> > Client checks if returned connectionId equals to the original and
>> > understands whether logical connection was restored or not.
>> >
>> > Thoughts?
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-16928
>> >
>> > On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <pt...@apache.org>
>> > wrote:
>> >
>> > > Andrey,
>> > >
>> > > > different connections on different client instances theoretically
>> > > > could generate an already existing connection ID
>> > >
>> > > Do you mean an UUID collision?
>> > >
>> > > On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org> wrote:
>> > >
>> > >> Igor,
>> > >>
>> > >> Thanks for the proposal.
>> > >>
>> > >> I understand that such a situation is almost impossible but "if
>> > >> anything can go wrong, it will". Does the protocol take into account
>> > >> that different connections on different client instances
>> theoretically
>> > >> could generate an already existing connection ID?
>> > >>
>> > >> Also, do I understand correctly that a server has enough information
>> > >> about client connections so it will be possible to observe a
>> > >> connections list on the server? It would be useful for cluster
>> > >> monitoring purposes.
>> > >>
>> > >> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org>
>> wrote:
>> > >> >
>> > >> > 1. I've tried to clarify IDs part;
>> > >> > 2. Maybe you are right, though in this case we'd need to use
>> > >> authentication
>> > >> > in ConnectionRestoreReq. Which sounds reasonable to me.
>> > >> >
>> > >> > Best Regards,
>> > >> > Igor
>> > >> >
>> > >> >
>> > >> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <
>> ptupitsyn@apache.org
>> > >
>> > >> > wrote:
>> > >> >
>> > >> > > Igor,
>> > >> > >
>> > >> > > The proposal looks good to me. Very detailed!
>> > >> > >
>> > >> > > A couple comments:
>> > >> > >
>> > >> > > 1. There is a bit of a term mixup with "Connection ID", "Node
>> ID",
>> > >> "Token"
>> > >> > > - can you please review those?
>> > >> > >
>> > >> > > 2. > The Connection ID should be generated using a proper secure
>> > >> algorithm
>> > >> > > (additional research is required here) to make sure an intruder
>> can
>> > >> not
>> > >> > > generate an existing Connection ID
>> > >> > > Not sure about the reasoning here. I think randomUUID() should be
>> > >> enough:
>> > >> > > - In the case of an unsecured cluster it does not matter, because
>> > >> anyone
>> > >> > > can do anything.
>> > >> > > - In the case of a secured cluster it does not matter, because
>> > >> > > authentication/authorization keeps intruders out.
>> > >> > >
>> > >> > >
>> > >> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <isapego@apache.org
>> >
>> > >> wrote:
>> > >> > >
>> > >> > > > Hi, Igniters
>> > >> > > >
>> > >> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The
>> main
>> > >> idea is
>> > >> > > to
>> > >> > > > define client lifecycle as well as core algorithms and
>> mechanisms
>> > >> used by
>> > >> > > > clients. This proposal can be used as a reference for
>> > >> implementation of a
>> > >> > > > new client for Ignite when dealing with such problems as:
>> > >> > > >
>> > >> > > > - Resolving of user-provided addresses;
>> > >> > > > - Initial connection to a cluster;
>> > >> > > > - Maintaining cluster connection;
>> > >> > > > - Connection recovery;
>> > >> > > > - Connection break handling.
>> > >> > > >
>> > >> > > > So take a look and let me know what you think guys.
>> > >> > > >
>> > >> > > > [1] -
>> > >> > > >
>> > >> > >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
>> > >> > > >
>> > >> > > > Best Regards,
>> > >> > > > Igor
>> > >> > > >
>> > >> > >
>> > >>
>> > >
>> >
>>
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
Another point:
With the current proposal, there is no way for the server to distinguish
a broken connection from a normal user-initiated connection close.
So in either case we'll wait for some configured timeout before cleaning up
session resources.
I think this is fine, at least for now.
On Thu, May 19, 2022 at 10:55 PM Igor Sapego <is...@apache.org> wrote:
> Andrey,
>
> 1. If a server generates a UUID that already exists it can check and just
> re-generate it straight away
> as a check is just a simple map lookup.
>
> 2. Well, yes. This proposal does not consider monitoring, but with current
> approach it will be definitely
> easier to implement it properly.
>
> Pavel,
>
> Wow, that's really cool that you've already started working on it.
>
> Regarding your proposal I like it - it will really make a procedure easier
> and faster for a client. I'll change
> the IEP accordingly.
>
> Best Regards,
> Igor
>
>
> On Thu, May 19, 2022 at 11:40 PM Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Igor,
> >
> > I've started working on the initial sessions implementation [1],
> > and I'd like to clarify the procedure of logical connection restore.
> >
> > > client in its turn tries to establish a new *node connection*
> > > and restore *logical connection *using ConnectionRestoreReq
> >
> > This implies either an additional request, or something that replaces
> > normal handshake.
> > However, we need to handle two cases here:
> > - Logical connection is restored
> > - Logical connection is not restored (timed out, server restarted, etc),
> in
> > which case we still establish the connection and use it.
> >
> > To account for the second case, we should always start with a regular
> > handshake.
> > I propose to add a nullable UUID field to the handshake request to cover
> > both cases:
> > - connectionId is null or not found on server: proceed with normal
> > handshake, return newly generated connectionId.
> > - connectionId is not null and found on server: restore logical
> connection,
> > return the same connectionId
> >
> > Client checks if returned connectionId equals to the original and
> > understands whether logical connection was restored or not.
> >
> > Thoughts?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-16928
> >
> > On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> > > Andrey,
> > >
> > > > different connections on different client instances theoretically
> > > > could generate an already existing connection ID
> > >
> > > Do you mean an UUID collision?
> > >
> > > On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org> wrote:
> > >
> > >> Igor,
> > >>
> > >> Thanks for the proposal.
> > >>
> > >> I understand that such a situation is almost impossible but "if
> > >> anything can go wrong, it will". Does the protocol take into account
> > >> that different connections on different client instances theoretically
> > >> could generate an already existing connection ID?
> > >>
> > >> Also, do I understand correctly that a server has enough information
> > >> about client connections so it will be possible to observe a
> > >> connections list on the server? It would be useful for cluster
> > >> monitoring purposes.
> > >>
> > >> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org>
> wrote:
> > >> >
> > >> > 1. I've tried to clarify IDs part;
> > >> > 2. Maybe you are right, though in this case we'd need to use
> > >> authentication
> > >> > in ConnectionRestoreReq. Which sounds reasonable to me.
> > >> >
> > >> > Best Regards,
> > >> > Igor
> > >> >
> > >> >
> > >> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <
> ptupitsyn@apache.org
> > >
> > >> > wrote:
> > >> >
> > >> > > Igor,
> > >> > >
> > >> > > The proposal looks good to me. Very detailed!
> > >> > >
> > >> > > A couple comments:
> > >> > >
> > >> > > 1. There is a bit of a term mixup with "Connection ID", "Node ID",
> > >> "Token"
> > >> > > - can you please review those?
> > >> > >
> > >> > > 2. > The Connection ID should be generated using a proper secure
> > >> algorithm
> > >> > > (additional research is required here) to make sure an intruder
> can
> > >> not
> > >> > > generate an existing Connection ID
> > >> > > Not sure about the reasoning here. I think randomUUID() should be
> > >> enough:
> > >> > > - In the case of an unsecured cluster it does not matter, because
> > >> anyone
> > >> > > can do anything.
> > >> > > - In the case of a secured cluster it does not matter, because
> > >> > > authentication/authorization keeps intruders out.
> > >> > >
> > >> > >
> > >> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > Hi, Igniters
> > >> > > >
> > >> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main
> > >> idea is
> > >> > > to
> > >> > > > define client lifecycle as well as core algorithms and
> mechanisms
> > >> used by
> > >> > > > clients. This proposal can be used as a reference for
> > >> implementation of a
> > >> > > > new client for Ignite when dealing with such problems as:
> > >> > > >
> > >> > > > - Resolving of user-provided addresses;
> > >> > > > - Initial connection to a cluster;
> > >> > > > - Maintaining cluster connection;
> > >> > > > - Connection recovery;
> > >> > > > - Connection break handling.
> > >> > > >
> > >> > > > So take a look and let me know what you think guys.
> > >> > > >
> > >> > > > [1] -
> > >> > > >
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> > >> > > >
> > >> > > > Best Regards,
> > >> > > > Igor
> > >> > > >
> > >> > >
> > >>
> > >
> >
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
Igor, ok, agreed.
On Mon, Jun 27, 2022 at 11:27 PM Igor Sapego <is...@apache.org> wrote:
> Pavel, let's postpone developing a sessions part until all the corner cases
> are clarified. I'll let you know when new the proposal is ready for
> discussion
> again.
>
> Best Regards,
> Igor
>
>
> On Mon, May 23, 2022 at 12:25 PM Andrey Gura <ag...@apache.org> wrote:
>
> > Thanks for the answers, Igor.
> >
> > On Thu, May 19, 2022 at 10:55 PM Igor Sapego <is...@apache.org> wrote:
> > >
> > > Andrey,
> > >
> > > 1. If a server generates a UUID that already exists it can check and
> just
> > > re-generate it straight away
> > > as a check is just a simple map lookup.
> > >
> > > 2. Well, yes. This proposal does not consider monitoring, but with
> > current
> > > approach it will be definitely
> > > easier to implement it properly.
> > >
> > > Pavel,
> > >
> > > Wow, that's really cool that you've already started working on it.
> > >
> > > Regarding your proposal I like it - it will really make a procedure
> > easier
> > > and faster for a client. I'll change
> > > the IEP accordingly.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Thu, May 19, 2022 at 11:40 PM Pavel Tupitsyn <pt...@apache.org>
> > > wrote:
> > >
> > > > Igor,
> > > >
> > > > I've started working on the initial sessions implementation [1],
> > > > and I'd like to clarify the procedure of logical connection restore.
> > > >
> > > > > client in its turn tries to establish a new *node connection*
> > > > > and restore *logical connection *using ConnectionRestoreReq
> > > >
> > > > This implies either an additional request, or something that replaces
> > > > normal handshake.
> > > > However, we need to handle two cases here:
> > > > - Logical connection is restored
> > > > - Logical connection is not restored (timed out, server restarted,
> > etc), in
> > > > which case we still establish the connection and use it.
> > > >
> > > > To account for the second case, we should always start with a regular
> > > > handshake.
> > > > I propose to add a nullable UUID field to the handshake request to
> > cover
> > > > both cases:
> > > > - connectionId is null or not found on server: proceed with normal
> > > > handshake, return newly generated connectionId.
> > > > - connectionId is not null and found on server: restore logical
> > connection,
> > > > return the same connectionId
> > > >
> > > > Client checks if returned connectionId equals to the original and
> > > > understands whether logical connection was restored or not.
> > > >
> > > > Thoughts?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-16928
> > > >
> > > > On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <
> ptupitsyn@apache.org>
> > > > wrote:
> > > >
> > > > > Andrey,
> > > > >
> > > > > > different connections on different client instances theoretically
> > > > > > could generate an already existing connection ID
> > > > >
> > > > > Do you mean an UUID collision?
> > > > >
> > > > > On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org>
> > wrote:
> > > > >
> > > > >> Igor,
> > > > >>
> > > > >> Thanks for the proposal.
> > > > >>
> > > > >> I understand that such a situation is almost impossible but "if
> > > > >> anything can go wrong, it will". Does the protocol take into
> account
> > > > >> that different connections on different client instances
> > theoretically
> > > > >> could generate an already existing connection ID?
> > > > >>
> > > > >> Also, do I understand correctly that a server has enough
> information
> > > > >> about client connections so it will be possible to observe a
> > > > >> connections list on the server? It would be useful for cluster
> > > > >> monitoring purposes.
> > > > >>
> > > > >> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org>
> > wrote:
> > > > >> >
> > > > >> > 1. I've tried to clarify IDs part;
> > > > >> > 2. Maybe you are right, though in this case we'd need to use
> > > > >> authentication
> > > > >> > in ConnectionRestoreReq. Which sounds reasonable to me.
> > > > >> >
> > > > >> > Best Regards,
> > > > >> > Igor
> > > > >> >
> > > > >> >
> > > > >> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Igor,
> > > > >> > >
> > > > >> > > The proposal looks good to me. Very detailed!
> > > > >> > >
> > > > >> > > A couple comments:
> > > > >> > >
> > > > >> > > 1. There is a bit of a term mixup with "Connection ID", "Node
> > ID",
> > > > >> "Token"
> > > > >> > > - can you please review those?
> > > > >> > >
> > > > >> > > 2. > The Connection ID should be generated using a proper
> secure
> > > > >> algorithm
> > > > >> > > (additional research is required here) to make sure an
> intruder
> > can
> > > > >> not
> > > > >> > > generate an existing Connection ID
> > > > >> > > Not sure about the reasoning here. I think randomUUID() should
> > be
> > > > >> enough:
> > > > >> > > - In the case of an unsecured cluster it does not matter,
> > because
> > > > >> anyone
> > > > >> > > can do anything.
> > > > >> > > - In the case of a secured cluster it does not matter, because
> > > > >> > > authentication/authorization keeps intruders out.
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <
> > isapego@apache.org>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Hi, Igniters
> > > > >> > > >
> > > > >> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The
> > main
> > > > >> idea is
> > > > >> > > to
> > > > >> > > > define client lifecycle as well as core algorithms and
> > mechanisms
> > > > >> used by
> > > > >> > > > clients. This proposal can be used as a reference for
> > > > >> implementation of a
> > > > >> > > > new client for Ignite when dealing with such problems as:
> > > > >> > > >
> > > > >> > > > - Resolving of user-provided addresses;
> > > > >> > > > - Initial connection to a cluster;
> > > > >> > > > - Maintaining cluster connection;
> > > > >> > > > - Connection recovery;
> > > > >> > > > - Connection break handling.
> > > > >> > > >
> > > > >> > > > So take a look and let me know what you think guys.
> > > > >> > > >
> > > > >> > > > [1] -
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> > > > >> > > >
> > > > >> > > > Best Regards,
> > > > >> > > > Igor
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > > >
> > > >
> >
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Igor Sapego <is...@apache.org>.
Pavel, let's postpone developing a sessions part until all the corner cases
are clarified. I'll let you know when new the proposal is ready for
discussion
again.
Best Regards,
Igor
On Mon, May 23, 2022 at 12:25 PM Andrey Gura <ag...@apache.org> wrote:
> Thanks for the answers, Igor.
>
> On Thu, May 19, 2022 at 10:55 PM Igor Sapego <is...@apache.org> wrote:
> >
> > Andrey,
> >
> > 1. If a server generates a UUID that already exists it can check and just
> > re-generate it straight away
> > as a check is just a simple map lookup.
> >
> > 2. Well, yes. This proposal does not consider monitoring, but with
> current
> > approach it will be definitely
> > easier to implement it properly.
> >
> > Pavel,
> >
> > Wow, that's really cool that you've already started working on it.
> >
> > Regarding your proposal I like it - it will really make a procedure
> easier
> > and faster for a client. I'll change
> > the IEP accordingly.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Thu, May 19, 2022 at 11:40 PM Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> > > Igor,
> > >
> > > I've started working on the initial sessions implementation [1],
> > > and I'd like to clarify the procedure of logical connection restore.
> > >
> > > > client in its turn tries to establish a new *node connection*
> > > > and restore *logical connection *using ConnectionRestoreReq
> > >
> > > This implies either an additional request, or something that replaces
> > > normal handshake.
> > > However, we need to handle two cases here:
> > > - Logical connection is restored
> > > - Logical connection is not restored (timed out, server restarted,
> etc), in
> > > which case we still establish the connection and use it.
> > >
> > > To account for the second case, we should always start with a regular
> > > handshake.
> > > I propose to add a nullable UUID field to the handshake request to
> cover
> > > both cases:
> > > - connectionId is null or not found on server: proceed with normal
> > > handshake, return newly generated connectionId.
> > > - connectionId is not null and found on server: restore logical
> connection,
> > > return the same connectionId
> > >
> > > Client checks if returned connectionId equals to the original and
> > > understands whether logical connection was restored or not.
> > >
> > > Thoughts?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-16928
> > >
> > > On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <pt...@apache.org>
> > > wrote:
> > >
> > > > Andrey,
> > > >
> > > > > different connections on different client instances theoretically
> > > > > could generate an already existing connection ID
> > > >
> > > > Do you mean an UUID collision?
> > > >
> > > > On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org>
> wrote:
> > > >
> > > >> Igor,
> > > >>
> > > >> Thanks for the proposal.
> > > >>
> > > >> I understand that such a situation is almost impossible but "if
> > > >> anything can go wrong, it will". Does the protocol take into account
> > > >> that different connections on different client instances
> theoretically
> > > >> could generate an already existing connection ID?
> > > >>
> > > >> Also, do I understand correctly that a server has enough information
> > > >> about client connections so it will be possible to observe a
> > > >> connections list on the server? It would be useful for cluster
> > > >> monitoring purposes.
> > > >>
> > > >> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org>
> wrote:
> > > >> >
> > > >> > 1. I've tried to clarify IDs part;
> > > >> > 2. Maybe you are right, though in this case we'd need to use
> > > >> authentication
> > > >> > in ConnectionRestoreReq. Which sounds reasonable to me.
> > > >> >
> > > >> > Best Regards,
> > > >> > Igor
> > > >> >
> > > >> >
> > > >> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <
> ptupitsyn@apache.org
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > Igor,
> > > >> > >
> > > >> > > The proposal looks good to me. Very detailed!
> > > >> > >
> > > >> > > A couple comments:
> > > >> > >
> > > >> > > 1. There is a bit of a term mixup with "Connection ID", "Node
> ID",
> > > >> "Token"
> > > >> > > - can you please review those?
> > > >> > >
> > > >> > > 2. > The Connection ID should be generated using a proper secure
> > > >> algorithm
> > > >> > > (additional research is required here) to make sure an intruder
> can
> > > >> not
> > > >> > > generate an existing Connection ID
> > > >> > > Not sure about the reasoning here. I think randomUUID() should
> be
> > > >> enough:
> > > >> > > - In the case of an unsecured cluster it does not matter,
> because
> > > >> anyone
> > > >> > > can do anything.
> > > >> > > - In the case of a secured cluster it does not matter, because
> > > >> > > authentication/authorization keeps intruders out.
> > > >> > >
> > > >> > >
> > > >> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <
> isapego@apache.org>
> > > >> wrote:
> > > >> > >
> > > >> > > > Hi, Igniters
> > > >> > > >
> > > >> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The
> main
> > > >> idea is
> > > >> > > to
> > > >> > > > define client lifecycle as well as core algorithms and
> mechanisms
> > > >> used by
> > > >> > > > clients. This proposal can be used as a reference for
> > > >> implementation of a
> > > >> > > > new client for Ignite when dealing with such problems as:
> > > >> > > >
> > > >> > > > - Resolving of user-provided addresses;
> > > >> > > > - Initial connection to a cluster;
> > > >> > > > - Maintaining cluster connection;
> > > >> > > > - Connection recovery;
> > > >> > > > - Connection break handling.
> > > >> > > >
> > > >> > > > So take a look and let me know what you think guys.
> > > >> > > >
> > > >> > > > [1] -
> > > >> > > >
> > > >> > >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> > > >> > > >
> > > >> > > > Best Regards,
> > > >> > > > Igor
> > > >> > > >
> > > >> > >
> > > >>
> > > >
> > >
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Andrey Gura <ag...@apache.org>.
Thanks for the answers, Igor.
On Thu, May 19, 2022 at 10:55 PM Igor Sapego <is...@apache.org> wrote:
>
> Andrey,
>
> 1. If a server generates a UUID that already exists it can check and just
> re-generate it straight away
> as a check is just a simple map lookup.
>
> 2. Well, yes. This proposal does not consider monitoring, but with current
> approach it will be definitely
> easier to implement it properly.
>
> Pavel,
>
> Wow, that's really cool that you've already started working on it.
>
> Regarding your proposal I like it - it will really make a procedure easier
> and faster for a client. I'll change
> the IEP accordingly.
>
> Best Regards,
> Igor
>
>
> On Thu, May 19, 2022 at 11:40 PM Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Igor,
> >
> > I've started working on the initial sessions implementation [1],
> > and I'd like to clarify the procedure of logical connection restore.
> >
> > > client in its turn tries to establish a new *node connection*
> > > and restore *logical connection *using ConnectionRestoreReq
> >
> > This implies either an additional request, or something that replaces
> > normal handshake.
> > However, we need to handle two cases here:
> > - Logical connection is restored
> > - Logical connection is not restored (timed out, server restarted, etc), in
> > which case we still establish the connection and use it.
> >
> > To account for the second case, we should always start with a regular
> > handshake.
> > I propose to add a nullable UUID field to the handshake request to cover
> > both cases:
> > - connectionId is null or not found on server: proceed with normal
> > handshake, return newly generated connectionId.
> > - connectionId is not null and found on server: restore logical connection,
> > return the same connectionId
> >
> > Client checks if returned connectionId equals to the original and
> > understands whether logical connection was restored or not.
> >
> > Thoughts?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-16928
> >
> > On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> > > Andrey,
> > >
> > > > different connections on different client instances theoretically
> > > > could generate an already existing connection ID
> > >
> > > Do you mean an UUID collision?
> > >
> > > On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org> wrote:
> > >
> > >> Igor,
> > >>
> > >> Thanks for the proposal.
> > >>
> > >> I understand that such a situation is almost impossible but "if
> > >> anything can go wrong, it will". Does the protocol take into account
> > >> that different connections on different client instances theoretically
> > >> could generate an already existing connection ID?
> > >>
> > >> Also, do I understand correctly that a server has enough information
> > >> about client connections so it will be possible to observe a
> > >> connections list on the server? It would be useful for cluster
> > >> monitoring purposes.
> > >>
> > >> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org> wrote:
> > >> >
> > >> > 1. I've tried to clarify IDs part;
> > >> > 2. Maybe you are right, though in this case we'd need to use
> > >> authentication
> > >> > in ConnectionRestoreReq. Which sounds reasonable to me.
> > >> >
> > >> > Best Regards,
> > >> > Igor
> > >> >
> > >> >
> > >> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <ptupitsyn@apache.org
> > >
> > >> > wrote:
> > >> >
> > >> > > Igor,
> > >> > >
> > >> > > The proposal looks good to me. Very detailed!
> > >> > >
> > >> > > A couple comments:
> > >> > >
> > >> > > 1. There is a bit of a term mixup with "Connection ID", "Node ID",
> > >> "Token"
> > >> > > - can you please review those?
> > >> > >
> > >> > > 2. > The Connection ID should be generated using a proper secure
> > >> algorithm
> > >> > > (additional research is required here) to make sure an intruder can
> > >> not
> > >> > > generate an existing Connection ID
> > >> > > Not sure about the reasoning here. I think randomUUID() should be
> > >> enough:
> > >> > > - In the case of an unsecured cluster it does not matter, because
> > >> anyone
> > >> > > can do anything.
> > >> > > - In the case of a secured cluster it does not matter, because
> > >> > > authentication/authorization keeps intruders out.
> > >> > >
> > >> > >
> > >> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > Hi, Igniters
> > >> > > >
> > >> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main
> > >> idea is
> > >> > > to
> > >> > > > define client lifecycle as well as core algorithms and mechanisms
> > >> used by
> > >> > > > clients. This proposal can be used as a reference for
> > >> implementation of a
> > >> > > > new client for Ignite when dealing with such problems as:
> > >> > > >
> > >> > > > - Resolving of user-provided addresses;
> > >> > > > - Initial connection to a cluster;
> > >> > > > - Maintaining cluster connection;
> > >> > > > - Connection recovery;
> > >> > > > - Connection break handling.
> > >> > > >
> > >> > > > So take a look and let me know what you think guys.
> > >> > > >
> > >> > > > [1] -
> > >> > > >
> > >> > >
> > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> > >> > > >
> > >> > > > Best Regards,
> > >> > > > Igor
> > >> > > >
> > >> > >
> > >>
> > >
> >
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Igor Sapego <is...@apache.org>.
Andrey,
1. If a server generates a UUID that already exists it can check and just
re-generate it straight away
as a check is just a simple map lookup.
2. Well, yes. This proposal does not consider monitoring, but with current
approach it will be definitely
easier to implement it properly.
Pavel,
Wow, that's really cool that you've already started working on it.
Regarding your proposal I like it - it will really make a procedure easier
and faster for a client. I'll change
the IEP accordingly.
Best Regards,
Igor
On Thu, May 19, 2022 at 11:40 PM Pavel Tupitsyn <pt...@apache.org>
wrote:
> Igor,
>
> I've started working on the initial sessions implementation [1],
> and I'd like to clarify the procedure of logical connection restore.
>
> > client in its turn tries to establish a new *node connection*
> > and restore *logical connection *using ConnectionRestoreReq
>
> This implies either an additional request, or something that replaces
> normal handshake.
> However, we need to handle two cases here:
> - Logical connection is restored
> - Logical connection is not restored (timed out, server restarted, etc), in
> which case we still establish the connection and use it.
>
> To account for the second case, we should always start with a regular
> handshake.
> I propose to add a nullable UUID field to the handshake request to cover
> both cases:
> - connectionId is null or not found on server: proceed with normal
> handshake, return newly generated connectionId.
> - connectionId is not null and found on server: restore logical connection,
> return the same connectionId
>
> Client checks if returned connectionId equals to the original and
> understands whether logical connection was restored or not.
>
> Thoughts?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-16928
>
> On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Andrey,
> >
> > > different connections on different client instances theoretically
> > > could generate an already existing connection ID
> >
> > Do you mean an UUID collision?
> >
> > On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org> wrote:
> >
> >> Igor,
> >>
> >> Thanks for the proposal.
> >>
> >> I understand that such a situation is almost impossible but "if
> >> anything can go wrong, it will". Does the protocol take into account
> >> that different connections on different client instances theoretically
> >> could generate an already existing connection ID?
> >>
> >> Also, do I understand correctly that a server has enough information
> >> about client connections so it will be possible to observe a
> >> connections list on the server? It would be useful for cluster
> >> monitoring purposes.
> >>
> >> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org> wrote:
> >> >
> >> > 1. I've tried to clarify IDs part;
> >> > 2. Maybe you are right, though in this case we'd need to use
> >> authentication
> >> > in ConnectionRestoreReq. Which sounds reasonable to me.
> >> >
> >> > Best Regards,
> >> > Igor
> >> >
> >> >
> >> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <ptupitsyn@apache.org
> >
> >> > wrote:
> >> >
> >> > > Igor,
> >> > >
> >> > > The proposal looks good to me. Very detailed!
> >> > >
> >> > > A couple comments:
> >> > >
> >> > > 1. There is a bit of a term mixup with "Connection ID", "Node ID",
> >> "Token"
> >> > > - can you please review those?
> >> > >
> >> > > 2. > The Connection ID should be generated using a proper secure
> >> algorithm
> >> > > (additional research is required here) to make sure an intruder can
> >> not
> >> > > generate an existing Connection ID
> >> > > Not sure about the reasoning here. I think randomUUID() should be
> >> enough:
> >> > > - In the case of an unsecured cluster it does not matter, because
> >> anyone
> >> > > can do anything.
> >> > > - In the case of a secured cluster it does not matter, because
> >> > > authentication/authorization keeps intruders out.
> >> > >
> >> > >
> >> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org>
> >> wrote:
> >> > >
> >> > > > Hi, Igniters
> >> > > >
> >> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main
> >> idea is
> >> > > to
> >> > > > define client lifecycle as well as core algorithms and mechanisms
> >> used by
> >> > > > clients. This proposal can be used as a reference for
> >> implementation of a
> >> > > > new client for Ignite when dealing with such problems as:
> >> > > >
> >> > > > - Resolving of user-provided addresses;
> >> > > > - Initial connection to a cluster;
> >> > > > - Maintaining cluster connection;
> >> > > > - Connection recovery;
> >> > > > - Connection break handling.
> >> > > >
> >> > > > So take a look and let me know what you think guys.
> >> > > >
> >> > > > [1] -
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> >> > > >
> >> > > > Best Regards,
> >> > > > Igor
> >> > > >
> >> > >
> >>
> >
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
Igor,
I've started working on the initial sessions implementation [1],
and I'd like to clarify the procedure of logical connection restore.
> client in its turn tries to establish a new *node connection*
> and restore *logical connection *using ConnectionRestoreReq
This implies either an additional request, or something that replaces
normal handshake.
However, we need to handle two cases here:
- Logical connection is restored
- Logical connection is not restored (timed out, server restarted, etc), in
which case we still establish the connection and use it.
To account for the second case, we should always start with a regular
handshake.
I propose to add a nullable UUID field to the handshake request to cover
both cases:
- connectionId is null or not found on server: proceed with normal
handshake, return newly generated connectionId.
- connectionId is not null and found on server: restore logical connection,
return the same connectionId
Client checks if returned connectionId equals to the original and
understands whether logical connection was restored or not.
Thoughts?
[1] https://issues.apache.org/jira/browse/IGNITE-16928
On Thu, May 19, 2022 at 10:27 PM Pavel Tupitsyn <pt...@apache.org>
wrote:
> Andrey,
>
> > different connections on different client instances theoretically
> > could generate an already existing connection ID
>
> Do you mean an UUID collision?
>
> On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org> wrote:
>
>> Igor,
>>
>> Thanks for the proposal.
>>
>> I understand that such a situation is almost impossible but "if
>> anything can go wrong, it will". Does the protocol take into account
>> that different connections on different client instances theoretically
>> could generate an already existing connection ID?
>>
>> Also, do I understand correctly that a server has enough information
>> about client connections so it will be possible to observe a
>> connections list on the server? It would be useful for cluster
>> monitoring purposes.
>>
>> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org> wrote:
>> >
>> > 1. I've tried to clarify IDs part;
>> > 2. Maybe you are right, though in this case we'd need to use
>> authentication
>> > in ConnectionRestoreReq. Which sounds reasonable to me.
>> >
>> > Best Regards,
>> > Igor
>> >
>> >
>> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <pt...@apache.org>
>> > wrote:
>> >
>> > > Igor,
>> > >
>> > > The proposal looks good to me. Very detailed!
>> > >
>> > > A couple comments:
>> > >
>> > > 1. There is a bit of a term mixup with "Connection ID", "Node ID",
>> "Token"
>> > > - can you please review those?
>> > >
>> > > 2. > The Connection ID should be generated using a proper secure
>> algorithm
>> > > (additional research is required here) to make sure an intruder can
>> not
>> > > generate an existing Connection ID
>> > > Not sure about the reasoning here. I think randomUUID() should be
>> enough:
>> > > - In the case of an unsecured cluster it does not matter, because
>> anyone
>> > > can do anything.
>> > > - In the case of a secured cluster it does not matter, because
>> > > authentication/authorization keeps intruders out.
>> > >
>> > >
>> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org>
>> wrote:
>> > >
>> > > > Hi, Igniters
>> > > >
>> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main
>> idea is
>> > > to
>> > > > define client lifecycle as well as core algorithms and mechanisms
>> used by
>> > > > clients. This proposal can be used as a reference for
>> implementation of a
>> > > > new client for Ignite when dealing with such problems as:
>> > > >
>> > > > - Resolving of user-provided addresses;
>> > > > - Initial connection to a cluster;
>> > > > - Maintaining cluster connection;
>> > > > - Connection recovery;
>> > > > - Connection break handling.
>> > > >
>> > > > So take a look and let me know what you think guys.
>> > > >
>> > > > [1] -
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
>> > > >
>> > > > Best Regards,
>> > > > Igor
>> > > >
>> > >
>>
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
Andrey,
> different connections on different client instances theoretically
> could generate an already existing connection ID
Do you mean an UUID collision?
On Thu, May 19, 2022 at 1:09 AM Andrey Gura <ag...@apache.org> wrote:
> Igor,
>
> Thanks for the proposal.
>
> I understand that such a situation is almost impossible but "if
> anything can go wrong, it will". Does the protocol take into account
> that different connections on different client instances theoretically
> could generate an already existing connection ID?
>
> Also, do I understand correctly that a server has enough information
> about client connections so it will be possible to observe a
> connections list on the server? It would be useful for cluster
> monitoring purposes.
>
> On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org> wrote:
> >
> > 1. I've tried to clarify IDs part;
> > 2. Maybe you are right, though in this case we'd need to use
> authentication
> > in ConnectionRestoreReq. Which sounds reasonable to me.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <pt...@apache.org>
> > wrote:
> >
> > > Igor,
> > >
> > > The proposal looks good to me. Very detailed!
> > >
> > > A couple comments:
> > >
> > > 1. There is a bit of a term mixup with "Connection ID", "Node ID",
> "Token"
> > > - can you please review those?
> > >
> > > 2. > The Connection ID should be generated using a proper secure
> algorithm
> > > (additional research is required here) to make sure an intruder can not
> > > generate an existing Connection ID
> > > Not sure about the reasoning here. I think randomUUID() should be
> enough:
> > > - In the case of an unsecured cluster it does not matter, because
> anyone
> > > can do anything.
> > > - In the case of a secured cluster it does not matter, because
> > > authentication/authorization keeps intruders out.
> > >
> > >
> > > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org>
> wrote:
> > >
> > > > Hi, Igniters
> > > >
> > > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main
> idea is
> > > to
> > > > define client lifecycle as well as core algorithms and mechanisms
> used by
> > > > clients. This proposal can be used as a reference for implementation
> of a
> > > > new client for Ignite when dealing with such problems as:
> > > >
> > > > - Resolving of user-provided addresses;
> > > > - Initial connection to a cluster;
> > > > - Maintaining cluster connection;
> > > > - Connection recovery;
> > > > - Connection break handling.
> > > >
> > > > So take a look and let me know what you think guys.
> > > >
> > > > [1] -
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > >
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Andrey Gura <ag...@apache.org>.
Igor,
Thanks for the proposal.
I understand that such a situation is almost impossible but "if
anything can go wrong, it will". Does the protocol take into account
that different connections on different client instances theoretically
could generate an already existing connection ID?
Also, do I understand correctly that a server has enough information
about client connections so it will be possible to observe a
connections list on the server? It would be useful for cluster
monitoring purposes.
On Tue, May 17, 2022 at 3:11 PM Igor Sapego <is...@apache.org> wrote:
>
> 1. I've tried to clarify IDs part;
> 2. Maybe you are right, though in this case we'd need to use authentication
> in ConnectionRestoreReq. Which sounds reasonable to me.
>
> Best Regards,
> Igor
>
>
> On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> > Igor,
> >
> > The proposal looks good to me. Very detailed!
> >
> > A couple comments:
> >
> > 1. There is a bit of a term mixup with "Connection ID", "Node ID", "Token"
> > - can you please review those?
> >
> > 2. > The Connection ID should be generated using a proper secure algorithm
> > (additional research is required here) to make sure an intruder can not
> > generate an existing Connection ID
> > Not sure about the reasoning here. I think randomUUID() should be enough:
> > - In the case of an unsecured cluster it does not matter, because anyone
> > can do anything.
> > - In the case of a secured cluster it does not matter, because
> > authentication/authorization keeps intruders out.
> >
> >
> > On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org> wrote:
> >
> > > Hi, Igniters
> > >
> > > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main idea is
> > to
> > > define client lifecycle as well as core algorithms and mechanisms used by
> > > clients. This proposal can be used as a reference for implementation of a
> > > new client for Ignite when dealing with such problems as:
> > >
> > > - Resolving of user-provided addresses;
> > > - Initial connection to a cluster;
> > > - Maintaining cluster connection;
> > > - Connection recovery;
> > > - Connection break handling.
> > >
> > > So take a look and let me know what you think guys.
> > >
> > > [1] -
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> > >
> > > Best Regards,
> > > Igor
> > >
> >
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Igor Sapego <is...@apache.org>.
1. I've tried to clarify IDs part;
2. Maybe you are right, though in this case we'd need to use authentication
in ConnectionRestoreReq. Which sounds reasonable to me.
Best Regards,
Igor
On Tue, May 17, 2022 at 10:47 AM Pavel Tupitsyn <pt...@apache.org>
wrote:
> Igor,
>
> The proposal looks good to me. Very detailed!
>
> A couple comments:
>
> 1. There is a bit of a term mixup with "Connection ID", "Node ID", "Token"
> - can you please review those?
>
> 2. > The Connection ID should be generated using a proper secure algorithm
> (additional research is required here) to make sure an intruder can not
> generate an existing Connection ID
> Not sure about the reasoning here. I think randomUUID() should be enough:
> - In the case of an unsecured cluster it does not matter, because anyone
> can do anything.
> - In the case of a secured cluster it does not matter, because
> authentication/authorization keeps intruders out.
>
>
> On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org> wrote:
>
> > Hi, Igniters
> >
> > I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main idea is
> to
> > define client lifecycle as well as core algorithms and mechanisms used by
> > clients. This proposal can be used as a reference for implementation of a
> > new client for Ignite when dealing with such problems as:
> >
> > - Resolving of user-provided addresses;
> > - Initial connection to a cluster;
> > - Maintaining cluster connection;
> > - Connection recovery;
> > - Connection break handling.
> >
> > So take a look and let me know what you think guys.
> >
> > [1] -
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
> >
> > Best Regards,
> > Igor
> >
>
Re: IEP-90 Ignite 3 Client Lifecycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
Igor,
The proposal looks good to me. Very detailed!
A couple comments:
1. There is a bit of a term mixup with "Connection ID", "Node ID", "Token"
- can you please review those?
2. > The Connection ID should be generated using a proper secure algorithm
(additional research is required here) to make sure an intruder can not
generate an existing Connection ID
Not sure about the reasoning here. I think randomUUID() should be enough:
- In the case of an unsecured cluster it does not matter, because anyone
can do anything.
- In the case of a secured cluster it does not matter, because
authentication/authorization keeps intruders out.
On Mon, May 16, 2022 at 11:07 PM Igor Sapego <is...@apache.org> wrote:
> Hi, Igniters
>
> I've prepared an IEP for Ignite 3 Client Lifecycle [1]. The main idea is to
> define client lifecycle as well as core algorithms and mechanisms used by
> clients. This proposal can be used as a reference for implementation of a
> new client for Ignite when dealing with such problems as:
>
> - Resolving of user-provided addresses;
> - Initial connection to a cluster;
> - Maintaining cluster connection;
> - Connection recovery;
> - Connection break handling.
>
> So take a look and let me know what you think guys.
>
> [1] -
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle
>
> Best Regards,
> Igor
>