You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alex Plehanov <pl...@gmail.com> on 2019/10/16 07:34:31 UTC
Re: Best Effort Affinity for thin clients
Hello guys,
I've implemented affinity awareness support for java thin client [1]. There
is only client-side affected by the patch. Can anyone review the change?
1: https://issues.apache.org/jira/browse/IGNITE-11898
ср, 13 мар. 2019 г. в 22:54, Pavel Tupitsyn <pt...@apache.org>:
> Default value for boolean is false, and I though we'll have the feature
> enabled by default.
> But I agree with you. Let's disable by default and name the config property
> EnableAffinityAwareness.
>
> On Wed, Mar 13, 2019 at 12:52 PM Igor Sapego <is...@apache.org> wrote:
>
> > For the "false" I mean "disable" here.
> >
> > BTW, I believe we should name this parameter in a positive way:
> > EnableAffinityAwareness, not disable.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Wed, Mar 13, 2019 at 12:50 PM Igor Sapego <is...@apache.org> wrote:
> >
> > > Well, yes, this looks like a simplest solution. Let's implement it for
> > the
> > > beginning and set this feature to "false" by default, as this feature
> > looks
> > > complex, probably error-prone, and should be considered in a "beta"
> > > state for the first release.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Mon, Mar 11, 2019 at 8:04 PM Pavel Tupitsyn <pt...@apache.org>
> > > wrote:
> > >
> > >> My suggestion is a boolean flag in client configuration:
> > >> DisableAffinityAwareness
> > >> And use old random/round-robin behavior with only one active
> connection.
> > >>
> > >> On Mon, Mar 11, 2019 at 1:36 PM Igor Sapego <is...@apache.org>
> wrote:
> > >>
> > >> > Pavel,
> > >> >
> > >> > That's right. Do you have other suggestions or objections?
> > >> >
> > >> > Best Regards,
> > >> > Igor
> > >> >
> > >> >
> > >> > On Fri, Mar 8, 2019 at 11:37 AM Pavel Tupitsyn <
> ptupitsyn@apache.org>
> > >> > wrote:
> > >> >
> > >> > > > maxConnectionNumber parameter
> > >> > > What's the idea? Follow the Best Effor Affinity logic, but
> establish
> > >> up
> > >> > to
> > >> > > N connections?
> > >> > >
> > >> > > On Thu, Mar 7, 2019 at 1:23 PM Igor Sapego <is...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > I can propose two improvements here:
> > >> > > >
> > >> > > > 1. A simple one. Lets introduce maxConnectionNumber parameter
> > >> > > > in ClientConfiguration. As it is easy to implement it may be
> > >> introduced
> > >> > > > together with the new feature to give user an additional
> control.
> > >> > > >
> > >> > > > 2. Asynchronous connection establishment. In this case startup
> > >> method
> > >> > > > of a client returns control to user once it have established at
> > >> least
> > >> > one
> > >> > > > connection. Other connections established in background by a
> > >> separate
> > >> > > > thread. This one is harder to implement and maybe it makes sense
> > to
> > >> add
> > >> > > > it as a separate feature.
> > >> > > >
> > >> > > > Best Regards,
> > >> > > > Igor
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Mar 6, 2019 at 9:43 PM Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I'm in progress of implementing this IEP for Ignite.NET, and I
> > >> have
> > >> > > > > concerns about the following:
> > >> > > > >
> > >> > > > > > On thin client startup it connects to all nodes provided by
> > >> client
> > >> > > > > configuration
> > >> > > > >
> > >> > > > > Should we, at least, make this behavior optional?
> > >> > > > >
> > >> > > > > One of the benefits of thin client is quick startup/connect
> time
> > >> and
> > >> > > low
> > >> > > > > resource usage.
> > >> > > > > Adding "connect all" behavior can negate those benefits,
> > >> especially
> > >> > on
> > >> > > > > large clusters.
> > >> > > > >
> > >> > > > > Thoughts?
> > >> > > > >
> > >> > > > > On Thu, Feb 14, 2019 at 5:39 PM Igor Sapego <
> isapego@apache.org
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Guys, I've updated the IEP page [1] once again.
> > >> > > > > >
> > >> > > > > > Please, pay attention to sections Cache affinity mapping
> > >> acquiring
> > >> > > > > > (4.a, format of Cache Partitions Request) and Changes to
> cache
> > >> > > > > > operations with single key (3 and 4, algorithm).
> > >> > > > > >
> > >> > > > > > Long story short, I've decided to add some additional data
> to
> > >> Cache
> > >> > > > > > Partitions Response, so that client can determine how to
> > >> calculate
> > >> > > > > > partition for a given key properly.
> > >> > > > > >
> > >> > > > > > [1] -
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > >> > > > > >
> > >> > > > > > Best Regards,
> > >> > > > > > Igor
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Mon, Feb 4, 2019 at 8:24 PM Pavel Tupitsyn <
> > >> > ptupitsyn@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Looks good to me.
> > >> > > > > > >
> > >> > > > > > > On Mon, Feb 4, 2019 at 6:30 PM Igor Sapego <
> > >> isapego@apache.org>
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > I've updated IEP page: [1]
> > >> > > > > > > >
> > >> > > > > > > > What do you think now? To me it looks cleaner.
> > >> > > > > > > >
> > >> > > > > > > > [1] -
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > >> > > > > > > >
> > >> > > > > > > > Best Regards,
> > >> > > > > > > > Igor
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Mon, Feb 4, 2019 at 4:44 PM Igor Sapego <
> > >> isapego@apache.org
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Ok, I understand now. I'll try updating IEP according
> to
> > >> this
> > >> > > > > > proposal
> > >> > > > > > > > and
> > >> > > > > > > > > notify you guys.
> > >> > > > > > > > >
> > >> > > > > > > > > Best Regards,
> > >> > > > > > > > > Igor
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Mon, Feb 4, 2019 at 4:27 PM Vladimir Ozerov <
> > >> > > > > vozerov@gridgain.com
> > >> > > > > > >
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Igor,
> > >> > > > > > > > >>
> > >> > > > > > > > >> My idea is simply to add the list of caches with the
> > same
> > >> > > > > > distribution
> > >> > > > > > > > to
> > >> > > > > > > > >> the end of partition response. Client can use this
> > >> > information
> > >> > > > to
> > >> > > > > > > > populate
> > >> > > > > > > > >> partition info for more caches in a single request.
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Mon, Feb 4, 2019 at 3:06 PM Igor Sapego <
> > >> > > isapego@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >> > Vladimir,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > So correct me if I'm wrong, what you propose is to
> > >> avoid
> > >> > > > > > mentioning
> > >> > > > > > > > >> > of cache groups, and use instead of "cache group"
> > term
> > >> > > > something
> > >> > > > > > > like
> > >> > > > > > > > >> > "distribution"? Or do you propose some changes in
> > >> > protocol?
> > >> > > If
> > >> > > > > so,
> > >> > > > > > > can
> > >> > > > > > > > >> > you briefly explain, what kind of changes they are?
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Best Regards,
> > >> > > > > > > > >> > Igor
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > On Mon, Feb 4, 2019 at 1:13 PM Vladimir Ozerov <
> > >> > > > > > > vozerov@gridgain.com>
> > >> > > > > > > > >> > wrote:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > > Igor,
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > Yes, cache groups are public API. However, we try
> > to
> > >> > avoid
> > >> > > > new
> > >> > > > > > > APIs
> > >> > > > > > > > >> > > depending on them.
> > >> > > > > > > > >> > > The main point from my side is that “similar
> cache
> > >> > group”
> > >> > > > can
> > >> > > > > be
> > >> > > > > > > > >> easily
> > >> > > > > > > > >> > > generalized to “similar distribution”. This way
> we
> > >> avoid
> > >> > > > cache
> > >> > > > > > > > groups
> > >> > > > > > > > >> on
> > >> > > > > > > > >> > > protocol level at virtually no cost.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > Vladimir.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > пн, 4 февр. 2019 г. в 12:48, Igor Sapego <
> > >> > > > isapego@apache.org
> > >> > > > > >:
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > > Guys,
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > Can you explain why do we want to avoid Cache
> > >> groups
> > >> > in
> > >> > > > > > > protocol?
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > If it's about simplicity of the protocol, then
> > >> > removing
> > >> > > > > cache
> > >> > > > > > > > groups
> > >> > > > > > > > >> > will
> > >> > > > > > > > >> > > > not help much with it - we will still need to
> > >> include
> > >> > > > > > > > >> "knownCacheIds"
> > >> > > > > > > > >> > > > field in request and
> > >> "cachesWithTheSamePartitioning"
> > >> > > field
> > >> > > > > in
> > >> > > > > > > > >> response.
> > >> > > > > > > > >> > > > And also, since when do Ignite prefers
> simplicity
> > >> over
> > >> > > > > > > > performance?
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > If it's about not wanting to show internals of
> > >> Ignite
> > >> > > then
> > >> > > > > it
> > >> > > > > > > > sounds
> > >> > > > > > > > >> > like
> > >> > > > > > > > >> > > > a very weak argument to me, since Cache Groups
> > is a
> > >> > > public
> > >> > > > > > thing
> > >> > > > > > > > >> [1].
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > [1] -
> > >> > https://apacheignite.readme.io/docs/cache-groups
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > Best Regards,
> > >> > > > > > > > >> > > > Igor
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > On Mon, Feb 4, 2019 at 11:47 AM Vladimir
> Ozerov <
> > >> > > > > > > > >> vozerov@gridgain.com>
> > >> > > > > > > > >> > > > wrote:
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > > Pavel, Igor,
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > This is not very accurate to say that this
> will
> > >> not
> > >> > > save
> > >> > > > > > > memory.
> > >> > > > > > > > >> In
> > >> > > > > > > > >> > > > > practice we observed a number of OOME issues
> on
> > >> the
> > >> > > > > > > server-side
> > >> > > > > > > > >> due
> > >> > > > > > > > >> > to
> > >> > > > > > > > >> > > > many
> > >> > > > > > > > >> > > > > caches and it was one of motivations for
> cache
> > >> > groups
> > >> > > > > > (another
> > >> > > > > > > > one
> > >> > > > > > > > >> > disk
> > >> > > > > > > > >> > > > > access optimizations). On the other hand, I
> > agree
> > >> > that
> > >> > > > > we'd
> > >> > > > > > > > >> better to
> > >> > > > > > > > >> > > > avoid
> > >> > > > > > > > >> > > > > cache groups in the protocol because this is
> > >> > internal
> > >> > > > > > > > >> implementation
> > >> > > > > > > > >> > > > detail
> > >> > > > > > > > >> > > > > which is likely (I hope so) to be changed in
> > >> future.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > So I have another proposal - let's track
> caches
> > >> with
> > >> > > the
> > >> > > > > > same
> > >> > > > > > > > >> > affinity
> > >> > > > > > > > >> > > > > distribution instead. That is, normally most
> of
> > >> > > > > PARTITIONED
> > >> > > > > > > > caches
> > >> > > > > > > > >> > will
> > >> > > > > > > > >> > > > > have very few variants of configuration: it
> > will
> > >> be
> > >> > > > > > Rendezvous
> > >> > > > > > > > >> > affinity
> > >> > > > > > > > >> > > > > function, most likely with default partition
> > >> number
> > >> > > and
> > >> > > > > with
> > >> > > > > > > 1-2
> > >> > > > > > > > >> > > backups
> > >> > > > > > > > >> > > > at
> > >> > > > > > > > >> > > > > most. So when affinity distribution for
> > specific
> > >> > cache
> > >> > > > is
> > >> > > > > > > > >> requested,
> > >> > > > > > > > >> > we
> > >> > > > > > > > >> > > > can
> > >> > > > > > > > >> > > > > append to the response *list of caches with
> the
> > >> same
> > >> > > > > > > > >> distribution*.
> > >> > > > > > > > >> > > I.e.:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > class AffinityResponse {
> > >> > > > > > > > >> > > > > Object distribution; // Actual
> > >> distribution
> > >> > > > > > > > >> > > > > List<Integer> cacheIds; // Caches with
> > >> similar
> > >> > > > > > > distribution
> > >> > > > > > > > >> > > > > }
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Makes sense?
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > On Sun, Feb 3, 2019 at 8:31 PM Pavel
> Tupitsyn <
> > >> > > > > > > > >> ptupitsyn@apache.org>
> > >> > > > > > > > >> > > > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > Igor, I have a feeling that we should omit
> > >> Cache
> > >> > > Group
> > >> > > > > > stuff
> > >> > > > > > > > >> from
> > >> > > > > > > > >> > the
> > >> > > > > > > > >> > > > > > protocol.
> > >> > > > > > > > >> > > > > > It is a rare use case and even then dealing
> > >> with
> > >> > > them
> > >> > > > on
> > >> > > > > > > > client
> > >> > > > > > > > >> > > barely
> > >> > > > > > > > >> > > > > > saves some memory.
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > We can keep it simple and have partition
> map
> > >> per
> > >> > > > > cacheId.
> > >> > > > > > > > >> Thoughts?
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > On Fri, Feb 1, 2019 at 6:49 PM Igor Sapego
> <
> > >> > > > > > > > isapego@apache.org>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > Guys, I've updated the proposal once
> again
> > >> [1],
> > >> > so
> > >> > > > > > please,
> > >> > > > > > > > >> > > > > > > take a look and let me know what you
> think.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > [1] -
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Best Regards,
> > >> > > > > > > > >> > > > > > > Igor
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > On Thu, Jan 17, 2019 at 1:05 PM Igor
> > Sapego <
> > >> > > > > > > > >> isapego@apache.org>
> > >> > > > > > > > >> > > > > wrote:
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > > Yeah, I'll add it.
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Best Regards,
> > >> > > > > > > > >> > > > > > > > Igor
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > On Wed, Jan 16, 2019 at 11:08 PM Pavel
> > >> > Tupitsyn
> > >> > > <
> > >> > > > > > > > >> > > > > ptupitsyn@apache.org>
> > >> > > > > > > > >> > > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > to every server
> > >> > > > > > > > >> > > > > > > >> I did not think of this issue. Now I
> > agree
> > >> > with
> > >> > > > > your
> > >> > > > > > > > >> approach.
> > >> > > > > > > > >> > > > > > > >> Can you please add an explanation of
> > this
> > >> to
> > >> > > the
> > >> > > > > IEP?
> > >> > > > > > > > >> > > > > > > >>
> > >> > > > > > > > >> > > > > > > >> Thanks!
> > >> > > > > > > > >> > > > > > > >>
> > >> > > > > > > > >> > > > > > > >> On Wed, Jan 16, 2019 at 2:53 PM Igor
> > >> Sapego <
> > >> > > > > > > > >> > isapego@apache.org
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >>
> > >> > > > > > > > >> > > > > > > >> > Pavel,
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >> > Yeah, it makes sense, but to me it
> > seems
> > >> > that
> > >> > > > > this
> > >> > > > > > > > >> approach
> > >> > > > > > > > >> > > can
> > >> > > > > > > > >> > > > > lead
> > >> > > > > > > > >> > > > > > > >> > to more complicated client logic, as
> > it
> > >> > will
> > >> > > > > > require
> > >> > > > > > > to
> > >> > > > > > > > >> make
> > >> > > > > > > > >> > > > > > > additional
> > >> > > > > > > > >> > > > > > > >> > call
> > >> > > > > > > > >> > > > > > > >> > to every server, that reports
> affinity
> > >> > > topology
> > >> > > > > > > change.
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >> > Guys, WDYT?
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >> > Best Regards,
> > >> > > > > > > > >> > > > > > > >> > Igor
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >> > On Tue, Jan 15, 2019 at 10:59 PM
> Pavel
> > >> > > > Tupitsyn <
> > >> > > > > > > > >> > > > > > ptupitsyn@apache.org
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > wrote:
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >> > > Igor,
> > >> > > > > > > > >> > > > > > > >> > >
> > >> > > > > > > > >> > > > > > > >> > > > It is proposed to add flag to
> > every
> > >> > > > > response,
> > >> > > > > > > that
> > >> > > > > > > > >> > shows
> > >> > > > > > > > >> > > > > > whether
> > >> > > > > > > > >> > > > > > > >> the
> > >> > > > > > > > >> > > > > > > >> > > Affinity Topology Version of the
> > >> cluster
> > >> > > has
> > >> > > > > > > changed
> > >> > > > > > > > >> since
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > last
> > >> > > > > > > > >> > > > > > > >> > request
> > >> > > > > > > > >> > > > > > > >> > > from the client.
> > >> > > > > > > > >> > > > > > > >> > > I propose to keep this flag. So no
> > >> need
> > >> > for
> > >> > > > > > > periodic
> > >> > > > > > > > >> > checks.
> > >> > > > > > > > >> > > > > Makes
> > >> > > > > > > > >> > > > > > > >> sense?
> > >> > > > > > > > >> > > > > > > >> > >
> > >> > > > > > > > >> > > > > > > >> > > On Tue, Jan 15, 2019 at 4:45 PM
> Igor
> > >> > > Sapego <
> > >> > > > > > > > >> > > > isapego@apache.org
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> wrote:
> > >> > > > > > > > >> > > > > > > >> > >
> > >> > > > > > > > >> > > > > > > >> > > > Pavel,
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > This will require from client to
> > >> send
> > >> > > this
> > >> > > > > new
> > >> > > > > > > > >> request
> > >> > > > > > > > >> > > > > > > periodically,
> > >> > > > > > > > >> > > > > > > >> > I'm
> > >> > > > > > > > >> > > > > > > >> > > > not
> > >> > > > > > > > >> > > > > > > >> > > > sure this will make clients
> > simpler.
> > >> > > > Anyway,
> > >> > > > > > > let's
> > >> > > > > > > > >> > discuss
> > >> > > > > > > > >> > > > it.
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > Vladimir,
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > With current proposal, we will
> > have
> > >> > > > affinity
> > >> > > > > > info
> > >> > > > > > > > in
> > >> > > > > > > > >> > > message
> > >> > > > > > > > >> > > > > > > header.
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > Best Regards,
> > >> > > > > > > > >> > > > > > > >> > > > Igor
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > On Tue, Jan 15, 2019 at 11:01 AM
> > >> > Vladimir
> > >> > > > > > Ozerov
> > >> > > > > > > <
> > >> > > > > > > > >> > > > > > > >> vozerov@gridgain.com
> > >> > > > > > > > >> > > > > > > >> > >
> > >> > > > > > > > >> > > > > > > >> > > > wrote:
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > > Igor,
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > I think that "Cache Partitions
> > >> > Request"
> > >> > > > > > should
> > >> > > > > > > > >> contain
> > >> > > > > > > > >> > > > > > affinity
> > >> > > > > > > > >> > > > > > > >> > > topology
> > >> > > > > > > > >> > > > > > > >> > > > > version. Otherwise we do not
> > know
> > >> > what
> > >> > > > > > > > >> distribution is
> > >> > > > > > > > >> > > > > > returned
> > >> > > > > > > > >> > > > > > > -
> > >> > > > > > > > >> > > > > > > >> the
> > >> > > > > > > > >> > > > > > > >> > > one
> > >> > > > > > > > >> > > > > > > >> > > > > we expected, or some newer
> one.
> > >> The
> > >> > > > latter
> > >> > > > > > may
> > >> > > > > > > > >> happen
> > >> > > > > > > > >> > in
> > >> > > > > > > > >> > > > > case
> > >> > > > > > > > >> > > > > > > >> > topology
> > >> > > > > > > > >> > > > > > > >> > > > > changed or late affinity
> > >> assignment
> > >> > > > > happened
> > >> > > > > > > > >> between
> > >> > > > > > > > >> > > > server
> > >> > > > > > > > >> > > > > > > >> response
> > >> > > > > > > > >> > > > > > > >> > > and
> > >> > > > > > > > >> > > > > > > >> > > > > subsequent client partitions
> > >> request.
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > Vladimir.
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > On Mon, Jan 14, 2019 at 6:08
> PM
> > >> Igor
> > >> > > > > Sapego <
> > >> > > > > > > > >> > > > > > isapego@apache.org
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > Hello guys,
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > I've updated IEP page [1]
> > >> > describing
> > >> > > > > > proposed
> > >> > > > > > > > >> > solution
> > >> > > > > > > > >> > > > in
> > >> > > > > > > > >> > > > > > more
> > >> > > > > > > > >> > > > > > > >> > > details
> > >> > > > > > > > >> > > > > > > >> > > > > and
> > >> > > > > > > > >> > > > > > > >> > > > > > proposing some changes for a
> > >> > > protocol.
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > Please, take a look and let
> me
> > >> know
> > >> > > > what
> > >> > > > > > you
> > >> > > > > > > > >> think.
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > [1] -
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > >
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >>
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > Best Regards,
> > >> > > > > > > > >> > > > > > > >> > > > > > Igor
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > On Tue, Jun 19, 2018 at
> 11:54
> > AM
> > >> > > > Vladimir
> > >> > > > > > > > Ozerov
> > >> > > > > > > > >> <
> > >> > > > > > > > >> > > > > > > >> > > vozerov@gridgain.com
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > Denis,
> > >> > > > > > > > >> > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > Yes, in principle we can
> > >> extend
> > >> > it.
> > >> > > > We
> > >> > > > > > are
> > >> > > > > > > > >> going
> > >> > > > > > > > >> > to
> > >> > > > > > > > >> > > > > > > implement
> > >> > > > > > > > >> > > > > > > >> it
> > >> > > > > > > > >> > > > > > > >> > in
> > >> > > > > > > > >> > > > > > > >> > > > > > > subsequent phases of this
> > IEP.
> > >> > > > > > > > >> > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > On Tue, Jun 19, 2018 at
> 4:30
> > >> AM,
> > >> > > > > Dmitriy
> > >> > > > > > > > >> > Setrakyan <
> > >> > > > > > > > >> > > > > > > >> > > > > > dsetrakyan@apache.org>
> > >> > > > > > > > >> > > > > > > >> > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > > On Mon, Jun 18, 2018 at
> > >> 11:07
> > >> > AM,
> > >> > > > > Denis
> > >> > > > > > > > >> Magda <
> > >> > > > > > > > >> > > > > > > >> > dmagda@apache.org
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > > > Folks,
> > >> > > > > > > > >> > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > > > Feel that this
> > >> functionality
> > >> > > can
> > >> > > > be
> > >> > > > > > > > >> extended
> > >> > > > > > > > >> > to
> > >> > > > > > > > >> > > > the
> > >> > > > > > > > >> > > > > > > >> automatic
> > >> > > > > > > > >> > > > > > > >> > > > > > > reconnect,
> > >> > > > > > > > >> > > > > > > >> > > > > > > > > can't it? Presently we
> > >> > require
> > >> > > to
> > >> > > > > > > > provide a
> > >> > > > > > > > >> > > static
> > >> > > > > > > > >> > > > > > list
> > >> > > > > > > > >> > > > > > > of
> > >> > > > > > > > >> > > > > > > >> > IPs
> > >> > > > > > > > >> > > > > > > >> > > to
> > >> > > > > > > > >> > > > > > > >> > > > > be
> > >> > > > > > > > >> > > > > > > >> > > > > > > used
> > >> > > > > > > > >> > > > > > > >> > > > > > > > > at a reconnect time.
> By
> > >> > having
> > >> > > a
> > >> > > > > > > > partition
> > >> > > > > > > > >> map
> > >> > > > > > > > >> > > of
> > >> > > > > > > > >> > > > > all
> > >> > > > > > > > >> > > > > > > the
> > >> > > > > > > > >> > > > > > > >> > > nodes,
> > >> > > > > > > > >> > > > > > > >> > > > > the
> > >> > > > > > > > >> > > > > > > >> > > > > > > thin
> > >> > > > > > > > >> > > > > > > >> > > > > > > > > client should be able
> to
> > >> > > automate
> > >> > > > > > this
> > >> > > > > > > > >> piece.
> > >> > > > > > > > >> > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > > Not sure if static IP
> list
> > >> can
> > >> > be
> > >> > > > > > > avoided.
> > >> > > > > > > > >> What
> > >> > > > > > > > >> > > Igor
> > >> > > > > > > > >> > > > > is
> > >> > > > > > > > >> > > > > > > >> > > suggesting
> > >> > > > > > > > >> > > > > > > >> > > > is
> > >> > > > > > > > >> > > > > > > >> > > > > > > that
> > >> > > > > > > > >> > > > > > > >> > > > > > > > we try to pick the best
> > node
> > >> > out
> > >> > > of
> > >> > > > > the
> > >> > > > > > > > >> static
> > >> > > > > > > > >> > IP
> > >> > > > > > > > >> > > > > list.
> > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > > > D.
> > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > > >> > >
> > >> > > > > > > > >> > > > > > > >> >
> > >> > > > > > > > >> > > > > > > >>
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Best Effort Affinity for thin clients
Posted by Pavel Tupitsyn <pt...@apache.org>.
Did a partial review, left one comment on the PR.
We need another pair of eyes on this for sure.
On Wed, Oct 16, 2019 at 10:34 AM Alex Plehanov <pl...@gmail.com>
wrote:
> Hello guys,
>
>
> I've implemented affinity awareness support for java thin client [1]. There
> is only client-side affected by the patch. Can anyone review the change?
>
>
> 1: https://issues.apache.org/jira/browse/IGNITE-11898
>
>
>
> ср, 13 мар. 2019 г. в 22:54, Pavel Tupitsyn <pt...@apache.org>:
>
> > Default value for boolean is false, and I though we'll have the feature
> > enabled by default.
> > But I agree with you. Let's disable by default and name the config
> property
> > EnableAffinityAwareness.
> >
> > On Wed, Mar 13, 2019 at 12:52 PM Igor Sapego <is...@apache.org> wrote:
> >
> > > For the "false" I mean "disable" here.
> > >
> > > BTW, I believe we should name this parameter in a positive way:
> > > EnableAffinityAwareness, not disable.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Wed, Mar 13, 2019 at 12:50 PM Igor Sapego <is...@apache.org>
> wrote:
> > >
> > > > Well, yes, this looks like a simplest solution. Let's implement it
> for
> > > the
> > > > beginning and set this feature to "false" by default, as this feature
> > > looks
> > > > complex, probably error-prone, and should be considered in a "beta"
> > > > state for the first release.
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > >
> > > > On Mon, Mar 11, 2019 at 8:04 PM Pavel Tupitsyn <ptupitsyn@apache.org
> >
> > > > wrote:
> > > >
> > > >> My suggestion is a boolean flag in client configuration:
> > > >> DisableAffinityAwareness
> > > >> And use old random/round-robin behavior with only one active
> > connection.
> > > >>
> > > >> On Mon, Mar 11, 2019 at 1:36 PM Igor Sapego <is...@apache.org>
> > wrote:
> > > >>
> > > >> > Pavel,
> > > >> >
> > > >> > That's right. Do you have other suggestions or objections?
> > > >> >
> > > >> > Best Regards,
> > > >> > Igor
> > > >> >
> > > >> >
> > > >> > On Fri, Mar 8, 2019 at 11:37 AM Pavel Tupitsyn <
> > ptupitsyn@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > > maxConnectionNumber parameter
> > > >> > > What's the idea? Follow the Best Effor Affinity logic, but
> > establish
> > > >> up
> > > >> > to
> > > >> > > N connections?
> > > >> > >
> > > >> > > On Thu, Mar 7, 2019 at 1:23 PM Igor Sapego <is...@apache.org>
> > > >> wrote:
> > > >> > >
> > > >> > > > I can propose two improvements here:
> > > >> > > >
> > > >> > > > 1. A simple one. Lets introduce maxConnectionNumber parameter
> > > >> > > > in ClientConfiguration. As it is easy to implement it may be
> > > >> introduced
> > > >> > > > together with the new feature to give user an additional
> > control.
> > > >> > > >
> > > >> > > > 2. Asynchronous connection establishment. In this case startup
> > > >> method
> > > >> > > > of a client returns control to user once it have established
> at
> > > >> least
> > > >> > one
> > > >> > > > connection. Other connections established in background by a
> > > >> separate
> > > >> > > > thread. This one is harder to implement and maybe it makes
> sense
> > > to
> > > >> add
> > > >> > > > it as a separate feature.
> > > >> > > >
> > > >> > > > Best Regards,
> > > >> > > > Igor
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Mar 6, 2019 at 9:43 PM Pavel Tupitsyn <
> > > ptupitsyn@apache.org
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi,
> > > >> > > > >
> > > >> > > > > I'm in progress of implementing this IEP for Ignite.NET,
> and I
> > > >> have
> > > >> > > > > concerns about the following:
> > > >> > > > >
> > > >> > > > > > On thin client startup it connects to all nodes provided
> by
> > > >> client
> > > >> > > > > configuration
> > > >> > > > >
> > > >> > > > > Should we, at least, make this behavior optional?
> > > >> > > > >
> > > >> > > > > One of the benefits of thin client is quick startup/connect
> > time
> > > >> and
> > > >> > > low
> > > >> > > > > resource usage.
> > > >> > > > > Adding "connect all" behavior can negate those benefits,
> > > >> especially
> > > >> > on
> > > >> > > > > large clusters.
> > > >> > > > >
> > > >> > > > > Thoughts?
> > > >> > > > >
> > > >> > > > > On Thu, Feb 14, 2019 at 5:39 PM Igor Sapego <
> > isapego@apache.org
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Guys, I've updated the IEP page [1] once again.
> > > >> > > > > >
> > > >> > > > > > Please, pay attention to sections Cache affinity mapping
> > > >> acquiring
> > > >> > > > > > (4.a, format of Cache Partitions Request) and Changes to
> > cache
> > > >> > > > > > operations with single key (3 and 4, algorithm).
> > > >> > > > > >
> > > >> > > > > > Long story short, I've decided to add some additional data
> > to
> > > >> Cache
> > > >> > > > > > Partitions Response, so that client can determine how to
> > > >> calculate
> > > >> > > > > > partition for a given key properly.
> > > >> > > > > >
> > > >> > > > > > [1] -
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > >> > > > > >
> > > >> > > > > > Best Regards,
> > > >> > > > > > Igor
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Mon, Feb 4, 2019 at 8:24 PM Pavel Tupitsyn <
> > > >> > ptupitsyn@apache.org>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Looks good to me.
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Feb 4, 2019 at 6:30 PM Igor Sapego <
> > > >> isapego@apache.org>
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > I've updated IEP page: [1]
> > > >> > > > > > > >
> > > >> > > > > > > > What do you think now? To me it looks cleaner.
> > > >> > > > > > > >
> > > >> > > > > > > > [1] -
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > >> > > > > > > >
> > > >> > > > > > > > Best Regards,
> > > >> > > > > > > > Igor
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Mon, Feb 4, 2019 at 4:44 PM Igor Sapego <
> > > >> isapego@apache.org
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Ok, I understand now. I'll try updating IEP
> according
> > to
> > > >> this
> > > >> > > > > > proposal
> > > >> > > > > > > > and
> > > >> > > > > > > > > notify you guys.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Best Regards,
> > > >> > > > > > > > > Igor
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Mon, Feb 4, 2019 at 4:27 PM Vladimir Ozerov <
> > > >> > > > > vozerov@gridgain.com
> > > >> > > > > > >
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > >> Igor,
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> My idea is simply to add the list of caches with
> the
> > > same
> > > >> > > > > > distribution
> > > >> > > > > > > > to
> > > >> > > > > > > > >> the end of partition response. Client can use this
> > > >> > information
> > > >> > > > to
> > > >> > > > > > > > populate
> > > >> > > > > > > > >> partition info for more caches in a single request.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On Mon, Feb 4, 2019 at 3:06 PM Igor Sapego <
> > > >> > > isapego@apache.org>
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> > Vladimir,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > So correct me if I'm wrong, what you propose is
> to
> > > >> avoid
> > > >> > > > > > mentioning
> > > >> > > > > > > > >> > of cache groups, and use instead of "cache group"
> > > term
> > > >> > > > something
> > > >> > > > > > > like
> > > >> > > > > > > > >> > "distribution"? Or do you propose some changes in
> > > >> > protocol?
> > > >> > > If
> > > >> > > > > so,
> > > >> > > > > > > can
> > > >> > > > > > > > >> > you briefly explain, what kind of changes they
> are?
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Best Regards,
> > > >> > > > > > > > >> > Igor
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > On Mon, Feb 4, 2019 at 1:13 PM Vladimir Ozerov <
> > > >> > > > > > > vozerov@gridgain.com>
> > > >> > > > > > > > >> > wrote:
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > > Igor,
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > Yes, cache groups are public API. However, we
> try
> > > to
> > > >> > avoid
> > > >> > > > new
> > > >> > > > > > > APIs
> > > >> > > > > > > > >> > > depending on them.
> > > >> > > > > > > > >> > > The main point from my side is that “similar
> > cache
> > > >> > group”
> > > >> > > > can
> > > >> > > > > be
> > > >> > > > > > > > >> easily
> > > >> > > > > > > > >> > > generalized to “similar distribution”. This way
> > we
> > > >> avoid
> > > >> > > > cache
> > > >> > > > > > > > groups
> > > >> > > > > > > > >> on
> > > >> > > > > > > > >> > > protocol level at virtually no cost.
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > Vladimir.
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > пн, 4 февр. 2019 г. в 12:48, Igor Sapego <
> > > >> > > > isapego@apache.org
> > > >> > > > > >:
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > > Guys,
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > Can you explain why do we want to avoid Cache
> > > >> groups
> > > >> > in
> > > >> > > > > > > protocol?
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > If it's about simplicity of the protocol,
> then
> > > >> > removing
> > > >> > > > > cache
> > > >> > > > > > > > groups
> > > >> > > > > > > > >> > will
> > > >> > > > > > > > >> > > > not help much with it - we will still need to
> > > >> include
> > > >> > > > > > > > >> "knownCacheIds"
> > > >> > > > > > > > >> > > > field in request and
> > > >> "cachesWithTheSamePartitioning"
> > > >> > > field
> > > >> > > > > in
> > > >> > > > > > > > >> response.
> > > >> > > > > > > > >> > > > And also, since when do Ignite prefers
> > simplicity
> > > >> over
> > > >> > > > > > > > performance?
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > If it's about not wanting to show internals
> of
> > > >> Ignite
> > > >> > > then
> > > >> > > > > it
> > > >> > > > > > > > sounds
> > > >> > > > > > > > >> > like
> > > >> > > > > > > > >> > > > a very weak argument to me, since Cache
> Groups
> > > is a
> > > >> > > public
> > > >> > > > > > thing
> > > >> > > > > > > > >> [1].
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > [1] -
> > > >> > https://apacheignite.readme.io/docs/cache-groups
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > Best Regards,
> > > >> > > > > > > > >> > > > Igor
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > On Mon, Feb 4, 2019 at 11:47 AM Vladimir
> > Ozerov <
> > > >> > > > > > > > >> vozerov@gridgain.com>
> > > >> > > > > > > > >> > > > wrote:
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > Pavel, Igor,
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > This is not very accurate to say that this
> > will
> > > >> not
> > > >> > > save
> > > >> > > > > > > memory.
> > > >> > > > > > > > >> In
> > > >> > > > > > > > >> > > > > practice we observed a number of OOME
> issues
> > on
> > > >> the
> > > >> > > > > > > server-side
> > > >> > > > > > > > >> due
> > > >> > > > > > > > >> > to
> > > >> > > > > > > > >> > > > many
> > > >> > > > > > > > >> > > > > caches and it was one of motivations for
> > cache
> > > >> > groups
> > > >> > > > > > (another
> > > >> > > > > > > > one
> > > >> > > > > > > > >> > disk
> > > >> > > > > > > > >> > > > > access optimizations). On the other hand, I
> > > agree
> > > >> > that
> > > >> > > > > we'd
> > > >> > > > > > > > >> better to
> > > >> > > > > > > > >> > > > avoid
> > > >> > > > > > > > >> > > > > cache groups in the protocol because this
> is
> > > >> > internal
> > > >> > > > > > > > >> implementation
> > > >> > > > > > > > >> > > > detail
> > > >> > > > > > > > >> > > > > which is likely (I hope so) to be changed
> in
> > > >> future.
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > So I have another proposal - let's track
> > caches
> > > >> with
> > > >> > > the
> > > >> > > > > > same
> > > >> > > > > > > > >> > affinity
> > > >> > > > > > > > >> > > > > distribution instead. That is, normally
> most
> > of
> > > >> > > > > PARTITIONED
> > > >> > > > > > > > caches
> > > >> > > > > > > > >> > will
> > > >> > > > > > > > >> > > > > have very few variants of configuration: it
> > > will
> > > >> be
> > > >> > > > > > Rendezvous
> > > >> > > > > > > > >> > affinity
> > > >> > > > > > > > >> > > > > function, most likely with default
> partition
> > > >> number
> > > >> > > and
> > > >> > > > > with
> > > >> > > > > > > 1-2
> > > >> > > > > > > > >> > > backups
> > > >> > > > > > > > >> > > > at
> > > >> > > > > > > > >> > > > > most. So when affinity distribution for
> > > specific
> > > >> > cache
> > > >> > > > is
> > > >> > > > > > > > >> requested,
> > > >> > > > > > > > >> > we
> > > >> > > > > > > > >> > > > can
> > > >> > > > > > > > >> > > > > append to the response *list of caches with
> > the
> > > >> same
> > > >> > > > > > > > >> distribution*.
> > > >> > > > > > > > >> > > I.e.:
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > class AffinityResponse {
> > > >> > > > > > > > >> > > > > Object distribution; // Actual
> > > >> distribution
> > > >> > > > > > > > >> > > > > List<Integer> cacheIds; // Caches with
> > > >> similar
> > > >> > > > > > > distribution
> > > >> > > > > > > > >> > > > > }
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > Makes sense?
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > On Sun, Feb 3, 2019 at 8:31 PM Pavel
> > Tupitsyn <
> > > >> > > > > > > > >> ptupitsyn@apache.org>
> > > >> > > > > > > > >> > > > > wrote:
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > Igor, I have a feeling that we should
> omit
> > > >> Cache
> > > >> > > Group
> > > >> > > > > > stuff
> > > >> > > > > > > > >> from
> > > >> > > > > > > > >> > the
> > > >> > > > > > > > >> > > > > > protocol.
> > > >> > > > > > > > >> > > > > > It is a rare use case and even then
> dealing
> > > >> with
> > > >> > > them
> > > >> > > > on
> > > >> > > > > > > > client
> > > >> > > > > > > > >> > > barely
> > > >> > > > > > > > >> > > > > > saves some memory.
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > We can keep it simple and have partition
> > map
> > > >> per
> > > >> > > > > cacheId.
> > > >> > > > > > > > >> Thoughts?
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > On Fri, Feb 1, 2019 at 6:49 PM Igor
> Sapego
> > <
> > > >> > > > > > > > isapego@apache.org>
> > > >> > > > > > > > >> > > wrote:
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > Guys, I've updated the proposal once
> > again
> > > >> [1],
> > > >> > so
> > > >> > > > > > please,
> > > >> > > > > > > > >> > > > > > > take a look and let me know what you
> > think.
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > [1] -
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > Best Regards,
> > > >> > > > > > > > >> > > > > > > Igor
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > On Thu, Jan 17, 2019 at 1:05 PM Igor
> > > Sapego <
> > > >> > > > > > > > >> isapego@apache.org>
> > > >> > > > > > > > >> > > > > wrote:
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > > Yeah, I'll add it.
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > > Best Regards,
> > > >> > > > > > > > >> > > > > > > > Igor
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > > On Wed, Jan 16, 2019 at 11:08 PM
> Pavel
> > > >> > Tupitsyn
> > > >> > > <
> > > >> > > > > > > > >> > > > > ptupitsyn@apache.org>
> > > >> > > > > > > > >> > > > > > > > wrote:
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > to every server
> > > >> > > > > > > > >> > > > > > > >> I did not think of this issue. Now I
> > > agree
> > > >> > with
> > > >> > > > > your
> > > >> > > > > > > > >> approach.
> > > >> > > > > > > > >> > > > > > > >> Can you please add an explanation of
> > > this
> > > >> to
> > > >> > > the
> > > >> > > > > IEP?
> > > >> > > > > > > > >> > > > > > > >>
> > > >> > > > > > > > >> > > > > > > >> Thanks!
> > > >> > > > > > > > >> > > > > > > >>
> > > >> > > > > > > > >> > > > > > > >> On Wed, Jan 16, 2019 at 2:53 PM Igor
> > > >> Sapego <
> > > >> > > > > > > > >> > isapego@apache.org
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > wrote:
> > > >> > > > > > > > >> > > > > > > >>
> > > >> > > > > > > > >> > > > > > > >> > Pavel,
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >> > Yeah, it makes sense, but to me it
> > > seems
> > > >> > that
> > > >> > > > > this
> > > >> > > > > > > > >> approach
> > > >> > > > > > > > >> > > can
> > > >> > > > > > > > >> > > > > lead
> > > >> > > > > > > > >> > > > > > > >> > to more complicated client logic,
> as
> > > it
> > > >> > will
> > > >> > > > > > require
> > > >> > > > > > > to
> > > >> > > > > > > > >> make
> > > >> > > > > > > > >> > > > > > > additional
> > > >> > > > > > > > >> > > > > > > >> > call
> > > >> > > > > > > > >> > > > > > > >> > to every server, that reports
> > affinity
> > > >> > > topology
> > > >> > > > > > > change.
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >> > Guys, WDYT?
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >> > Best Regards,
> > > >> > > > > > > > >> > > > > > > >> > Igor
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >> > On Tue, Jan 15, 2019 at 10:59 PM
> > Pavel
> > > >> > > > Tupitsyn <
> > > >> > > > > > > > >> > > > > > ptupitsyn@apache.org
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > wrote:
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >> > > Igor,
> > > >> > > > > > > > >> > > > > > > >> > >
> > > >> > > > > > > > >> > > > > > > >> > > > It is proposed to add flag to
> > > every
> > > >> > > > > response,
> > > >> > > > > > > that
> > > >> > > > > > > > >> > shows
> > > >> > > > > > > > >> > > > > > whether
> > > >> > > > > > > > >> > > > > > > >> the
> > > >> > > > > > > > >> > > > > > > >> > > Affinity Topology Version of the
> > > >> cluster
> > > >> > > has
> > > >> > > > > > > changed
> > > >> > > > > > > > >> since
> > > >> > > > > > > > >> > > the
> > > >> > > > > > > > >> > > > > > last
> > > >> > > > > > > > >> > > > > > > >> > request
> > > >> > > > > > > > >> > > > > > > >> > > from the client.
> > > >> > > > > > > > >> > > > > > > >> > > I propose to keep this flag. So
> no
> > > >> need
> > > >> > for
> > > >> > > > > > > periodic
> > > >> > > > > > > > >> > checks.
> > > >> > > > > > > > >> > > > > Makes
> > > >> > > > > > > > >> > > > > > > >> sense?
> > > >> > > > > > > > >> > > > > > > >> > >
> > > >> > > > > > > > >> > > > > > > >> > > On Tue, Jan 15, 2019 at 4:45 PM
> > Igor
> > > >> > > Sapego <
> > > >> > > > > > > > >> > > > isapego@apache.org
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> wrote:
> > > >> > > > > > > > >> > > > > > > >> > >
> > > >> > > > > > > > >> > > > > > > >> > > > Pavel,
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > This will require from client
> to
> > > >> send
> > > >> > > this
> > > >> > > > > new
> > > >> > > > > > > > >> request
> > > >> > > > > > > > >> > > > > > > periodically,
> > > >> > > > > > > > >> > > > > > > >> > I'm
> > > >> > > > > > > > >> > > > > > > >> > > > not
> > > >> > > > > > > > >> > > > > > > >> > > > sure this will make clients
> > > simpler.
> > > >> > > > Anyway,
> > > >> > > > > > > let's
> > > >> > > > > > > > >> > discuss
> > > >> > > > > > > > >> > > > it.
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > Vladimir,
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > With current proposal, we will
> > > have
> > > >> > > > affinity
> > > >> > > > > > info
> > > >> > > > > > > > in
> > > >> > > > > > > > >> > > message
> > > >> > > > > > > > >> > > > > > > header.
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > Best Regards,
> > > >> > > > > > > > >> > > > > > > >> > > > Igor
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > On Tue, Jan 15, 2019 at 11:01
> AM
> > > >> > Vladimir
> > > >> > > > > > Ozerov
> > > >> > > > > > > <
> > > >> > > > > > > > >> > > > > > > >> vozerov@gridgain.com
> > > >> > > > > > > > >> > > > > > > >> > >
> > > >> > > > > > > > >> > > > > > > >> > > > wrote:
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > Igor,
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > I think that "Cache
> Partitions
> > > >> > Request"
> > > >> > > > > > should
> > > >> > > > > > > > >> contain
> > > >> > > > > > > > >> > > > > > affinity
> > > >> > > > > > > > >> > > > > > > >> > > topology
> > > >> > > > > > > > >> > > > > > > >> > > > > version. Otherwise we do not
> > > know
> > > >> > what
> > > >> > > > > > > > >> distribution is
> > > >> > > > > > > > >> > > > > > returned
> > > >> > > > > > > > >> > > > > > > -
> > > >> > > > > > > > >> > > > > > > >> the
> > > >> > > > > > > > >> > > > > > > >> > > one
> > > >> > > > > > > > >> > > > > > > >> > > > > we expected, or some newer
> > one.
> > > >> The
> > > >> > > > latter
> > > >> > > > > > may
> > > >> > > > > > > > >> happen
> > > >> > > > > > > > >> > in
> > > >> > > > > > > > >> > > > > case
> > > >> > > > > > > > >> > > > > > > >> > topology
> > > >> > > > > > > > >> > > > > > > >> > > > > changed or late affinity
> > > >> assignment
> > > >> > > > > happened
> > > >> > > > > > > > >> between
> > > >> > > > > > > > >> > > > server
> > > >> > > > > > > > >> > > > > > > >> response
> > > >> > > > > > > > >> > > > > > > >> > > and
> > > >> > > > > > > > >> > > > > > > >> > > > > subsequent client partitions
> > > >> request.
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > Vladimir.
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > On Mon, Jan 14, 2019 at 6:08
> > PM
> > > >> Igor
> > > >> > > > > Sapego <
> > > >> > > > > > > > >> > > > > > isapego@apache.org
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > wrote:
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > Hello guys,
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > I've updated IEP page [1]
> > > >> > describing
> > > >> > > > > > proposed
> > > >> > > > > > > > >> > solution
> > > >> > > > > > > > >> > > > in
> > > >> > > > > > > > >> > > > > > more
> > > >> > > > > > > > >> > > > > > > >> > > details
> > > >> > > > > > > > >> > > > > > > >> > > > > and
> > > >> > > > > > > > >> > > > > > > >> > > > > > proposing some changes
> for a
> > > >> > > protocol.
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > Please, take a look and
> let
> > me
> > > >> know
> > > >> > > > what
> > > >> > > > > > you
> > > >> > > > > > > > >> think.
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > [1] -
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > >
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >>
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > Best Regards,
> > > >> > > > > > > > >> > > > > > > >> > > > > > Igor
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > On Tue, Jun 19, 2018 at
> > 11:54
> > > AM
> > > >> > > > Vladimir
> > > >> > > > > > > > Ozerov
> > > >> > > > > > > > >> <
> > > >> > > > > > > > >> > > > > > > >> > > vozerov@gridgain.com
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > wrote:
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > Denis,
> > > >> > > > > > > > >> > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > Yes, in principle we can
> > > >> extend
> > > >> > it.
> > > >> > > > We
> > > >> > > > > > are
> > > >> > > > > > > > >> going
> > > >> > > > > > > > >> > to
> > > >> > > > > > > > >> > > > > > > implement
> > > >> > > > > > > > >> > > > > > > >> it
> > > >> > > > > > > > >> > > > > > > >> > in
> > > >> > > > > > > > >> > > > > > > >> > > > > > > subsequent phases of
> this
> > > IEP.
> > > >> > > > > > > > >> > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > On Tue, Jun 19, 2018 at
> > 4:30
> > > >> AM,
> > > >> > > > > Dmitriy
> > > >> > > > > > > > >> > Setrakyan <
> > > >> > > > > > > > >> > > > > > > >> > > > > > dsetrakyan@apache.org>
> > > >> > > > > > > > >> > > > > > > >> > > > > > > wrote:
> > > >> > > > > > > > >> > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > On Mon, Jun 18, 2018
> at
> > > >> 11:07
> > > >> > AM,
> > > >> > > > > Denis
> > > >> > > > > > > > >> Magda <
> > > >> > > > > > > > >> > > > > > > >> > dmagda@apache.org
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > wrote:
> > > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > > Folks,
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > > Feel that this
> > > >> functionality
> > > >> > > can
> > > >> > > > be
> > > >> > > > > > > > >> extended
> > > >> > > > > > > > >> > to
> > > >> > > > > > > > >> > > > the
> > > >> > > > > > > > >> > > > > > > >> automatic
> > > >> > > > > > > > >> > > > > > > >> > > > > > > reconnect,
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > > can't it? Presently
> we
> > > >> > require
> > > >> > > to
> > > >> > > > > > > > provide a
> > > >> > > > > > > > >> > > static
> > > >> > > > > > > > >> > > > > > list
> > > >> > > > > > > > >> > > > > > > of
> > > >> > > > > > > > >> > > > > > > >> > IPs
> > > >> > > > > > > > >> > > > > > > >> > > to
> > > >> > > > > > > > >> > > > > > > >> > > > > be
> > > >> > > > > > > > >> > > > > > > >> > > > > > > used
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > > at a reconnect time.
> > By
> > > >> > having
> > > >> > > a
> > > >> > > > > > > > partition
> > > >> > > > > > > > >> map
> > > >> > > > > > > > >> > > of
> > > >> > > > > > > > >> > > > > all
> > > >> > > > > > > > >> > > > > > > the
> > > >> > > > > > > > >> > > > > > > >> > > nodes,
> > > >> > > > > > > > >> > > > > > > >> > > > > the
> > > >> > > > > > > > >> > > > > > > >> > > > > > > thin
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > > client should be
> able
> > to
> > > >> > > automate
> > > >> > > > > > this
> > > >> > > > > > > > >> piece.
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > Not sure if static IP
> > list
> > > >> can
> > > >> > be
> > > >> > > > > > > avoided.
> > > >> > > > > > > > >> What
> > > >> > > > > > > > >> > > Igor
> > > >> > > > > > > > >> > > > > is
> > > >> > > > > > > > >> > > > > > > >> > > suggesting
> > > >> > > > > > > > >> > > > > > > >> > > > is
> > > >> > > > > > > > >> > > > > > > >> > > > > > > that
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > we try to pick the
> best
> > > node
> > > >> > out
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > > > >> static
> > > >> > > > > > > > >> > IP
> > > >> > > > > > > > >> > > > > list.
> > > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > > > D.
> > > >> > > > > > > > >> > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > > >> > > >
> > > >> > > > > > > > >> > > > > > > >> > >
> > > >> > > > > > > > >> > > > > > > >> >
> > > >> > > > > > > > >> > > > > > > >>
> > > >> > > > > > > > >> > > > > > > >
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>