You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Moti Nisenson <mo...@gmail.com> on 2020/11/02 16:07:16 UTC
Custom Affinity Functions proposed for removal?
I saw at
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
that custom affinity functions are on the potential wishlist for removal.
The way we're using it's very critical that we be able to control the
placement of data quite precisely - as part of that we specify explicitly
the partition we want in the key, and then our affinity function uses that
(else delegating to default rendezvous). We don't need all the
abilities there, although I think that often others do.
This seems to me to be a case that the benefit of removing this is minimal
and could cause quite a lot of disruption to users.
Thanks!
Re: Custom Affinity Functions proposed for removal?
Posted by Raymond Wilson <ra...@trimble.com>.
We actually use different mappers assigned to different caches using the
same source key, eg:
public class Key
{
p,
x,
y
}
public class AffinityFunctionOne : IAffinityFunction
{
public int GetPartition(object key)
{
return PartitionHash((object as Key).p)
}
}
public class AffinityFunctionTwo : IAffinityFunction
{
public int GetPartition(object key)
{
return PartitionHash((object as Key).x, (object as Key).y)
}
}
On reviewing this code I see we do actually perform the AssignPartitions()
step as this is a required aspect of implementing the IAffinityFunction
interface, but this is not necessary for our application. This is just a
round robin assignment of partitions to nodes in
the AffinityFunctionContext.CurrentTopologySnapshot list of nodes
participating in the cache the key is for.
Raymond.
On Thu, Nov 5, 2020 at 4:54 AM mnk <mo...@gmail.com> wrote:
> For our case, we have a stateful instance associated with each user
> session,
> that actually handles user requests. That stateful instance is usually
> long-lived. In this case, we want the data for that instance to be stored
> on
> other nodes - at the dispatch stage, we choose a node to host the session
> such that it isn't in the primary or backup nodes for the selected
> partition. In the case we lose that node, we can always recover our state
> on
> a new node from the data stored.
>
> Short-lived case is less interesting... in this case we dispatch to the
> primary node and thus there is affinity for doing the replay.
>
> Due to limitations in how cache groups work (whereby new caches still
> result
> in PME), we have our own implementation there for storing logically
> separated data in the same cache - we then have a garbage cleanup phase
> which can evaluate expired/terminated sessions and clean out the cache.
> That
> process is all done locally - in other words, a primary node is able to
> determine which sessions need cleaning and clean out the data all locally.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
--
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com
<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
Re: Custom Affinity Functions proposed for removal?
Posted by mnk <mo...@gmail.com>.
For our case, we have a stateful instance associated with each user session,
that actually handles user requests. That stateful instance is usually
long-lived. In this case, we want the data for that instance to be stored on
other nodes - at the dispatch stage, we choose a node to host the session
such that it isn't in the primary or backup nodes for the selected
partition. In the case we lose that node, we can always recover our state on
a new node from the data stored.
Short-lived case is less interesting... in this case we dispatch to the
primary node and thus there is affinity for doing the replay.
Due to limitations in how cache groups work (whereby new caches still result
in PME), we have our own implementation there for storing logically
separated data in the same cache - we then have a garbage cleanup phase
which can evaluate expired/terminated sessions and clean out the cache. That
process is all done locally - in other words, a primary node is able to
determine which sessions need cleaning and clean out the data all locally.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Custom Affinity Functions proposed for removal?
Posted by Valentin Kulichenko <va...@gmail.com>.
I've created a ticket for this:
https://issues.apache.org/jira/browse/IGNITE-13671
-Val
On Wed, Nov 4, 2020 at 12:53 AM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Thanks, Raymond. So the reason why you couldn't use the @AffinityKeyMapped
> annotation or the CacheKeyConfiguration is that collocation is based on
> *two fields*, not just one field. Is my understanding correct?
>
> If that's the case, I believe it can be easily improved by providing ways
> to specify multiple fields for collocation. The fact that this scenario
> requires a custom affinity mapper is a usability issue in my view.
>
> -Val
>
> On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <ra...@trimble.com>
> wrote:
>
>> If I have a primary key vector like this <x><y><t>, where <x> & <y> can
>> have varying values for <t>, and I want all keys with the same <x> and <y>
>> to reside in the same partition for processing colocation requirements
>> then
>> I can't use the standard Ignite mapping to do this, I need to use a custom
>> mapper than uses just the <x> & <y> components.
>>
>>
>> On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
>> valentin.kulichenko@gmail.com> wrote:
>>
>> > Raymond,
>> >
>> > Thanks for the details. So it sounds like you have a custom affinity
>> > mapper, not affinity function. This makes things simpler, but I'm still
>> > failing to understand why standard mechanisms for collocation [1] didn't
>> > work for you. Could you please clarify?
>> >
>> > [1]
>> >
>> https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
>> >
>> > -Val
>> >
>> > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <
>> raymond_wilson@trimble.com
>> > >
>> > wrote:
>> >
>> > > In terms of Key -> Partition -> Node mapping, we provide customer
>> > affinity
>> > > mappers for the Key -> Partition stage an allow Ignite to map
>> partitions
>> > to
>> > > nodes.
>> > >
>> > > Our keys are structs with multiple fields forming a composite primary
>> > key,
>> > > parts of which are spatially identifying and parts contain other
>> > > characteristics. We want to ensure all related spatial elements
>> resolve
>> > to
>> > > the same partition and so will be placed on the same node for query
>> > > colocation purposes.
>> > >
>> > >
>> > >
>> > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
>> > > valentin.kulichenko@gmail.com> wrote:
>> > >
>> > > > Moti, Raymond,
>> > > >
>> > > > Could you please describe your use cases in more detail? What are
>> the
>> > > types
>> > > > used as cache keys? What is the custom logic that you use for
>> affinity
>> > > > mapping? What was the exact reason to customize versus using
>> built-in
>> > > > collocation mechanisms?
>> > > >
>> > > > Ultimately, I'm sure that custom affinity functions will go away
>> cause
>> > > they
>> > > > introduce multiple potential issues. However, I would definitely
>> like
>> > to
>> > > > make sure that your use cases are still supported in Ignite 3.0 via
>> > some
>> > > > other means.
>> > > >
>> > > > -Val
>> > > >
>> > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
>> > > > alexey.scherbakoff@gmail.com> wrote:
>> > > >
>> > > > > Thanks for the clarification.
>> > > > >
>> > > > > There was no intention to remove the customizable key to partition
>> > > > mapping.
>> > > > >
>> > > > > Difficulties arise when mapping partitions to nodes, so it's
>> > desirable
>> > > to
>> > > > > have internally tested implementation with a way to customize it's
>> > > > behavior
>> > > > > without additional coding on the user side.
>> > > > >
>> > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
>> > > raymond_wilson@trimble.com
>> > > > >:
>> > > > >
>> > > > > > Just to be clear, the affinity functions we are using convert
>> keys
>> > to
>> > > > > > partitions, we do not map partitions to nodes and leave that to
>> > > Ignite.
>> > > > > >
>> > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
>> > > > > > alexey.scherbakoff@gmail.com> wrote:
>> > > > > >
>> > > > > > > Hello.
>> > > > > > >
>> > > > > > > Custom affinity functions can cause weird bugs and data loss
>> if
>> > > > > > implemented
>> > > > > > > wrongly.
>> > > > > > > There is an intention to keep a backup filter based on user
>> > > > attributes
>> > > > > > > (with additional validation logic to ensure correctness) for
>> > > > > controllable
>> > > > > > > data placement.
>> > > > > > >
>> > > > > > > Can you describe more precisely why you had to implement
>> custom
>> > > > > affinity
>> > > > > > > functions and not resort to default rendezvous affinity +
>> backup
>> > > > > filter ?
>> > > > > > >
>> > > > > > >
>> > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
>> > > > > raymond_wilson@trimble.com
>> > > > > > >:
>> > > > > > >
>> > > > > > > > We also use custom affinity functions (vis the C# client).
>> > > > > > > >
>> > > > > > > > The wish list mentions use of a particular annotation
>> > > > > > > > (@CentralizedAffinityFunction):
>> > > > > > > > Is the wish to remove just this annotation, or the ability
>> to
>> > > > define
>> > > > > > > custom
>> > > > > > > > affinity functions at all?
>> > > > > > > >
>> > > > > > > > In our case we use affinity functions to ensure particular
>> > > > > distribution
>> > > > > > > of
>> > > > > > > > spatial data across a processing cluster to ensure even load
>> > > > > > management.
>> > > > > > > >
>> > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
>> > > > > moti.nisenson@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I saw at
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
>> > > > > > > > > that custom affinity functions are on the potential
>> wishlist
>> > > for
>> > > > > > > removal.
>> > > > > > > > > The way we're using it's very critical that we be able to
>> > > control
>> > > > > the
>> > > > > > > > > placement of data quite precisely - as part of that we
>> > specify
>> > > > > > > explicitly
>> > > > > > > > > the partition we want in the key, and then our affinity
>> > > function
>> > > > > uses
>> > > > > > > > that
>> > > > > > > > > (else delegating to default rendezvous). We don't need all
>> > the
>> > > > > > > > > abilities there, although I think that often others do.
>> > > > > > > > >
>> > > > > > > > > This seems to me to be a case that the benefit of removing
>> > this
>> > > > is
>> > > > > > > > minimal
>> > > > > > > > > and could cause quite a lot of disruption to users.
>> > > > > > > > >
>> > > > > > > > > Thanks!
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > <http://www.trimble.com/>
>> > > > > > > > Raymond Wilson
>> > > > > > > > Solution Architect, Civil Construction Software Systems
>> (CCSS)
>> > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
>> > > > > > > > +64-21-2013317 Mobile
>> > > > > > > > raymond_wilson@trimble.com
>> > > > > > > >
>> > > > > > > > <
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > > Alexei Scherbakov
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > <http://www.trimble.com/>
>> > > > > > Raymond Wilson
>> > > > > > Solution Architect, Civil Construction Software Systems (CCSS)
>> > > > > > 11 Birmingham Drive | Christchurch, New Zealand
>> > > > > > +64-21-2013317 Mobile
>> > > > > > raymond_wilson@trimble.com
>> > > > > >
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Best regards,
>> > > > > Alexei Scherbakov
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > <http://www.trimble.com/>
>> > > Raymond Wilson
>> > > Solution Architect, Civil Construction Software Systems (CCSS)
>> > > 11 Birmingham Drive | Christchurch, New Zealand
>> > > +64-21-2013317 Mobile
>> > > raymond_wilson@trimble.com
>> > >
>> > > <
>> > >
>> >
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> > > >
>> > >
>> >
>>
>>
>> --
>> <http://www.trimble.com/>
>> Raymond Wilson
>> Solution Architect, Civil Construction Software Systems (CCSS)
>> 11 Birmingham Drive | Christchurch, New Zealand
>> +64-21-2013317 Mobile
>> raymond_wilson@trimble.com
>>
>> <
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> >
>>
>
Re: Custom Affinity Functions proposed for removal?
Posted by Valentin Kulichenko <va...@gmail.com>.
Could you give an example of such mapping?
If that’s possible, it might also be very helpful to see the implementation
of your mapper. Looking at the code is often the best way to understand a
use case :)
-Val
On Wed, Nov 4, 2020 at 12:29 PM Raymond Wilson <ra...@trimble.com>
wrote:
> Actually, it's worse than that...
>
> We have more than one key -> partition mapping for the same key (part of a
> CQRS pattern we use).
>
> Aren't key affinity functions essentially an API in any event?
>
> Cheers,
> Raymond.
>
>
> On Wed, Nov 4, 2020 at 9:54 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Thanks, Raymond. So the reason why you couldn't use the
> @AffinityKeyMapped
> > annotation or the CacheKeyConfiguration is that collocation is based on
> > *two fields*, not just one field. Is my understanding correct?
> >
> > If that's the case, I believe it can be easily improved by providing ways
> > to specify multiple fields for collocation. The fact that this scenario
> > requires a custom affinity mapper is a usability issue in my view.
> >
> > -Val
> >
> > On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <
> raymond_wilson@trimble.com
> > >
> > wrote:
> >
> > > If I have a primary key vector like this <x><y><t>, where <x> & <y> can
> > > have varying values for <t>, and I want all keys with the same <x> and
> > <y>
> > > to reside in the same partition for processing colocation requirements
> > then
> > > I can't use the standard Ignite mapping to do this, I need to use a
> > custom
> > > mapper than uses just the <x> & <y> components.
> > >
> > >
> > > On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Raymond,
> > > >
> > > > Thanks for the details. So it sounds like you have a custom affinity
> > > > mapper, not affinity function. This makes things simpler, but I'm
> still
> > > > failing to understand why standard mechanisms for collocation [1]
> > didn't
> > > > work for you. Could you please clarify?
> > > >
> > > > [1]
> > > >
> > https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
> > > >
> > > > -Val
> > > >
> > > > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <
> > > raymond_wilson@trimble.com
> > > > >
> > > > wrote:
> > > >
> > > > > In terms of Key -> Partition -> Node mapping, we provide customer
> > > > affinity
> > > > > mappers for the Key -> Partition stage an allow Ignite to map
> > > partitions
> > > > to
> > > > > nodes.
> > > > >
> > > > > Our keys are structs with multiple fields forming a composite
> primary
> > > > key,
> > > > > parts of which are spatially identifying and parts contain other
> > > > > characteristics. We want to ensure all related spatial elements
> > resolve
> > > > to
> > > > > the same partition and so will be placed on the same node for query
> > > > > colocation purposes.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> wrote:
> > > > >
> > > > > > Moti, Raymond,
> > > > > >
> > > > > > Could you please describe your use cases in more detail? What are
> > the
> > > > > types
> > > > > > used as cache keys? What is the custom logic that you use for
> > > affinity
> > > > > > mapping? What was the exact reason to customize versus using
> > built-in
> > > > > > collocation mechanisms?
> > > > > >
> > > > > > Ultimately, I'm sure that custom affinity functions will go away
> > > cause
> > > > > they
> > > > > > introduce multiple potential issues. However, I would definitely
> > like
> > > > to
> > > > > > make sure that your use cases are still supported in Ignite 3.0
> via
> > > > some
> > > > > > other means.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks for the clarification.
> > > > > > >
> > > > > > > There was no intention to remove the customizable key to
> > partition
> > > > > > mapping.
> > > > > > >
> > > > > > > Difficulties arise when mapping partitions to nodes, so it's
> > > > desirable
> > > > > to
> > > > > > > have internally tested implementation with a way to customize
> > it's
> > > > > > behavior
> > > > > > > without additional coding on the user side.
> > > > > > >
> > > > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> > > > > raymond_wilson@trimble.com
> > > > > > >:
> > > > > > >
> > > > > > > > Just to be clear, the affinity functions we are using convert
> > > keys
> > > > to
> > > > > > > > partitions, we do not map partitions to nodes and leave that
> to
> > > > > Ignite.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hello.
> > > > > > > > >
> > > > > > > > > Custom affinity functions can cause weird bugs and data
> loss
> > if
> > > > > > > > implemented
> > > > > > > > > wrongly.
> > > > > > > > > There is an intention to keep a backup filter based on
> user
> > > > > > attributes
> > > > > > > > > (with additional validation logic to ensure correctness)
> for
> > > > > > > controllable
> > > > > > > > > data placement.
> > > > > > > > >
> > > > > > > > > Can you describe more precisely why you had to implement
> > custom
> > > > > > > affinity
> > > > > > > > > functions and not resort to default rendezvous affinity +
> > > backup
> > > > > > > filter ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > > > > > > raymond_wilson@trimble.com
> > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > We also use custom affinity functions (vis the C#
> client).
> > > > > > > > > >
> > > > > > > > > > The wish list mentions use of a particular annotation
> > > > > > > > > > (@CentralizedAffinityFunction):
> > > > > > > > > > Is the wish to remove just this annotation, or the
> ability
> > to
> > > > > > define
> > > > > > > > > custom
> > > > > > > > > > affinity functions at all?
> > > > > > > > > >
> > > > > > > > > > In our case we use affinity functions to ensure
> particular
> > > > > > > distribution
> > > > > > > > > of
> > > > > > > > > > spatial data across a processing cluster to ensure even
> > load
> > > > > > > > management.
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > > > > > > moti.nisenson@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I saw at
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > > > > > that custom affinity functions are on the potential
> > > wishlist
> > > > > for
> > > > > > > > > removal.
> > > > > > > > > > > The way we're using it's very critical that we be able
> to
> > > > > control
> > > > > > > the
> > > > > > > > > > > placement of data quite precisely - as part of that we
> > > > specify
> > > > > > > > > explicitly
> > > > > > > > > > > the partition we want in the key, and then our affinity
> > > > > function
> > > > > > > uses
> > > > > > > > > > that
> > > > > > > > > > > (else delegating to default rendezvous). We don't need
> > all
> > > > the
> > > > > > > > > > > abilities there, although I think that often others do.
> > > > > > > > > > >
> > > > > > > > > > > This seems to me to be a case that the benefit of
> > removing
> > > > this
> > > > > > is
> > > > > > > > > > minimal
> > > > > > > > > > > and could cause quite a lot of disruption to users.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > <http://www.trimble.com/>
> > > > > > > > > > Raymond Wilson
> > > > > > > > > > Solution Architect, Civil Construction Software Systems
> > > (CCSS)
> > > > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > > > > +64-21-2013317 Mobile
> > > > > > > > > > raymond_wilson@trimble.com
> > > > > > > > > >
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Alexei Scherbakov
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > <http://www.trimble.com/>
> > > > > > > > Raymond Wilson
> > > > > > > > Solution Architect, Civil Construction Software Systems
> (CCSS)
> > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > > +64-21-2013317 Mobile
> > > > > > > > raymond_wilson@trimble.com
> > > > > > > >
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Alexei Scherbakov
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > <http://www.trimble.com/>
> > > > > Raymond Wilson
> > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > +64-21-2013317 Mobile
> > > > > raymond_wilson@trimble.com
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > <http://www.trimble.com/>
> > > Raymond Wilson
> > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > 11 Birmingham Drive | Christchurch, New Zealand
> > > +64-21-2013317 Mobile
> > > raymond_wilson@trimble.com
> > >
> > > <
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > >
> > >
> >
>
>
> --
> <http://www.trimble.com/>
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> +64-21-2013317 Mobile
> raymond_wilson@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>
Re: Custom Affinity Functions proposed for removal?
Posted by Raymond Wilson <ra...@trimble.com>.
Actually, it's worse than that...
We have more than one key -> partition mapping for the same key (part of a
CQRS pattern we use).
Aren't key affinity functions essentially an API in any event?
Cheers,
Raymond.
On Wed, Nov 4, 2020 at 9:54 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Thanks, Raymond. So the reason why you couldn't use the @AffinityKeyMapped
> annotation or the CacheKeyConfiguration is that collocation is based on
> *two fields*, not just one field. Is my understanding correct?
>
> If that's the case, I believe it can be easily improved by providing ways
> to specify multiple fields for collocation. The fact that this scenario
> requires a custom affinity mapper is a usability issue in my view.
>
> -Val
>
> On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <raymond_wilson@trimble.com
> >
> wrote:
>
> > If I have a primary key vector like this <x><y><t>, where <x> & <y> can
> > have varying values for <t>, and I want all keys with the same <x> and
> <y>
> > to reside in the same partition for processing colocation requirements
> then
> > I can't use the standard Ignite mapping to do this, I need to use a
> custom
> > mapper than uses just the <x> & <y> components.
> >
> >
> > On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Raymond,
> > >
> > > Thanks for the details. So it sounds like you have a custom affinity
> > > mapper, not affinity function. This makes things simpler, but I'm still
> > > failing to understand why standard mechanisms for collocation [1]
> didn't
> > > work for you. Could you please clarify?
> > >
> > > [1]
> > >
> https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
> > >
> > > -Val
> > >
> > > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <
> > raymond_wilson@trimble.com
> > > >
> > > wrote:
> > >
> > > > In terms of Key -> Partition -> Node mapping, we provide customer
> > > affinity
> > > > mappers for the Key -> Partition stage an allow Ignite to map
> > partitions
> > > to
> > > > nodes.
> > > >
> > > > Our keys are structs with multiple fields forming a composite primary
> > > key,
> > > > parts of which are spatially identifying and parts contain other
> > > > characteristics. We want to ensure all related spatial elements
> resolve
> > > to
> > > > the same partition and so will be placed on the same node for query
> > > > colocation purposes.
> > > >
> > > >
> > > >
> > > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Moti, Raymond,
> > > > >
> > > > > Could you please describe your use cases in more detail? What are
> the
> > > > types
> > > > > used as cache keys? What is the custom logic that you use for
> > affinity
> > > > > mapping? What was the exact reason to customize versus using
> built-in
> > > > > collocation mechanisms?
> > > > >
> > > > > Ultimately, I'm sure that custom affinity functions will go away
> > cause
> > > > they
> > > > > introduce multiple potential issues. However, I would definitely
> like
> > > to
> > > > > make sure that your use cases are still supported in Ignite 3.0 via
> > > some
> > > > > other means.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > >
> > > > > > Thanks for the clarification.
> > > > > >
> > > > > > There was no intention to remove the customizable key to
> partition
> > > > > mapping.
> > > > > >
> > > > > > Difficulties arise when mapping partitions to nodes, so it's
> > > desirable
> > > > to
> > > > > > have internally tested implementation with a way to customize
> it's
> > > > > behavior
> > > > > > without additional coding on the user side.
> > > > > >
> > > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> > > > raymond_wilson@trimble.com
> > > > > >:
> > > > > >
> > > > > > > Just to be clear, the affinity functions we are using convert
> > keys
> > > to
> > > > > > > partitions, we do not map partitions to nodes and leave that to
> > > > Ignite.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hello.
> > > > > > > >
> > > > > > > > Custom affinity functions can cause weird bugs and data loss
> if
> > > > > > > implemented
> > > > > > > > wrongly.
> > > > > > > > There is an intention to keep a backup filter based on user
> > > > > attributes
> > > > > > > > (with additional validation logic to ensure correctness) for
> > > > > > controllable
> > > > > > > > data placement.
> > > > > > > >
> > > > > > > > Can you describe more precisely why you had to implement
> custom
> > > > > > affinity
> > > > > > > > functions and not resort to default rendezvous affinity +
> > backup
> > > > > > filter ?
> > > > > > > >
> > > > > > > >
> > > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > > > > > raymond_wilson@trimble.com
> > > > > > > >:
> > > > > > > >
> > > > > > > > > We also use custom affinity functions (vis the C# client).
> > > > > > > > >
> > > > > > > > > The wish list mentions use of a particular annotation
> > > > > > > > > (@CentralizedAffinityFunction):
> > > > > > > > > Is the wish to remove just this annotation, or the ability
> to
> > > > > define
> > > > > > > > custom
> > > > > > > > > affinity functions at all?
> > > > > > > > >
> > > > > > > > > In our case we use affinity functions to ensure particular
> > > > > > distribution
> > > > > > > > of
> > > > > > > > > spatial data across a processing cluster to ensure even
> load
> > > > > > > management.
> > > > > > > > >
> > > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > > > > > moti.nisenson@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I saw at
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > > > > that custom affinity functions are on the potential
> > wishlist
> > > > for
> > > > > > > > removal.
> > > > > > > > > > The way we're using it's very critical that we be able to
> > > > control
> > > > > > the
> > > > > > > > > > placement of data quite precisely - as part of that we
> > > specify
> > > > > > > > explicitly
> > > > > > > > > > the partition we want in the key, and then our affinity
> > > > function
> > > > > > uses
> > > > > > > > > that
> > > > > > > > > > (else delegating to default rendezvous). We don't need
> all
> > > the
> > > > > > > > > > abilities there, although I think that often others do.
> > > > > > > > > >
> > > > > > > > > > This seems to me to be a case that the benefit of
> removing
> > > this
> > > > > is
> > > > > > > > > minimal
> > > > > > > > > > and could cause quite a lot of disruption to users.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > <http://www.trimble.com/>
> > > > > > > > > Raymond Wilson
> > > > > > > > > Solution Architect, Civil Construction Software Systems
> > (CCSS)
> > > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > > > +64-21-2013317 Mobile
> > > > > > > > > raymond_wilson@trimble.com
> > > > > > > > >
> > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Alexei Scherbakov
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > <http://www.trimble.com/>
> > > > > > > Raymond Wilson
> > > > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > +64-21-2013317 Mobile
> > > > > > > raymond_wilson@trimble.com
> > > > > > >
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best regards,
> > > > > > Alexei Scherbakov
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > <http://www.trimble.com/>
> > > > Raymond Wilson
> > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > +64-21-2013317 Mobile
> > > > raymond_wilson@trimble.com
> > > >
> > > > <
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > >
> > > >
> > >
> >
> >
> > --
> > <http://www.trimble.com/>
> > Raymond Wilson
> > Solution Architect, Civil Construction Software Systems (CCSS)
> > 11 Birmingham Drive | Christchurch, New Zealand
> > +64-21-2013317 Mobile
> > raymond_wilson@trimble.com
> >
> > <
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > >
> >
>
--
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com
<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
Re: Custom Affinity Functions proposed for removal?
Posted by Valentin Kulichenko <va...@gmail.com>.
Thanks, Raymond. So the reason why you couldn't use the @AffinityKeyMapped
annotation or the CacheKeyConfiguration is that collocation is based on
*two fields*, not just one field. Is my understanding correct?
If that's the case, I believe it can be easily improved by providing ways
to specify multiple fields for collocation. The fact that this scenario
requires a custom affinity mapper is a usability issue in my view.
-Val
On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <ra...@trimble.com>
wrote:
> If I have a primary key vector like this <x><y><t>, where <x> & <y> can
> have varying values for <t>, and I want all keys with the same <x> and <y>
> to reside in the same partition for processing colocation requirements then
> I can't use the standard Ignite mapping to do this, I need to use a custom
> mapper than uses just the <x> & <y> components.
>
>
> On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Raymond,
> >
> > Thanks for the details. So it sounds like you have a custom affinity
> > mapper, not affinity function. This makes things simpler, but I'm still
> > failing to understand why standard mechanisms for collocation [1] didn't
> > work for you. Could you please clarify?
> >
> > [1]
> > https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
> >
> > -Val
> >
> > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <
> raymond_wilson@trimble.com
> > >
> > wrote:
> >
> > > In terms of Key -> Partition -> Node mapping, we provide customer
> > affinity
> > > mappers for the Key -> Partition stage an allow Ignite to map
> partitions
> > to
> > > nodes.
> > >
> > > Our keys are structs with multiple fields forming a composite primary
> > key,
> > > parts of which are spatially identifying and parts contain other
> > > characteristics. We want to ensure all related spatial elements resolve
> > to
> > > the same partition and so will be placed on the same node for query
> > > colocation purposes.
> > >
> > >
> > >
> > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Moti, Raymond,
> > > >
> > > > Could you please describe your use cases in more detail? What are the
> > > types
> > > > used as cache keys? What is the custom logic that you use for
> affinity
> > > > mapping? What was the exact reason to customize versus using built-in
> > > > collocation mechanisms?
> > > >
> > > > Ultimately, I'm sure that custom affinity functions will go away
> cause
> > > they
> > > > introduce multiple potential issues. However, I would definitely like
> > to
> > > > make sure that your use cases are still supported in Ignite 3.0 via
> > some
> > > > other means.
> > > >
> > > > -Val
> > > >
> > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > > > alexey.scherbakoff@gmail.com> wrote:
> > > >
> > > > > Thanks for the clarification.
> > > > >
> > > > > There was no intention to remove the customizable key to partition
> > > > mapping.
> > > > >
> > > > > Difficulties arise when mapping partitions to nodes, so it's
> > desirable
> > > to
> > > > > have internally tested implementation with a way to customize it's
> > > > behavior
> > > > > without additional coding on the user side.
> > > > >
> > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> > > raymond_wilson@trimble.com
> > > > >:
> > > > >
> > > > > > Just to be clear, the affinity functions we are using convert
> keys
> > to
> > > > > > partitions, we do not map partitions to nodes and leave that to
> > > Ignite.
> > > > > >
> > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > > >
> > > > > > > Hello.
> > > > > > >
> > > > > > > Custom affinity functions can cause weird bugs and data loss if
> > > > > > implemented
> > > > > > > wrongly.
> > > > > > > There is an intention to keep a backup filter based on user
> > > > attributes
> > > > > > > (with additional validation logic to ensure correctness) for
> > > > > controllable
> > > > > > > data placement.
> > > > > > >
> > > > > > > Can you describe more precisely why you had to implement custom
> > > > > affinity
> > > > > > > functions and not resort to default rendezvous affinity +
> backup
> > > > > filter ?
> > > > > > >
> > > > > > >
> > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > > > > raymond_wilson@trimble.com
> > > > > > >:
> > > > > > >
> > > > > > > > We also use custom affinity functions (vis the C# client).
> > > > > > > >
> > > > > > > > The wish list mentions use of a particular annotation
> > > > > > > > (@CentralizedAffinityFunction):
> > > > > > > > Is the wish to remove just this annotation, or the ability to
> > > > define
> > > > > > > custom
> > > > > > > > affinity functions at all?
> > > > > > > >
> > > > > > > > In our case we use affinity functions to ensure particular
> > > > > distribution
> > > > > > > of
> > > > > > > > spatial data across a processing cluster to ensure even load
> > > > > > management.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > > > > moti.nisenson@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I saw at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > > > that custom affinity functions are on the potential
> wishlist
> > > for
> > > > > > > removal.
> > > > > > > > > The way we're using it's very critical that we be able to
> > > control
> > > > > the
> > > > > > > > > placement of data quite precisely - as part of that we
> > specify
> > > > > > > explicitly
> > > > > > > > > the partition we want in the key, and then our affinity
> > > function
> > > > > uses
> > > > > > > > that
> > > > > > > > > (else delegating to default rendezvous). We don't need all
> > the
> > > > > > > > > abilities there, although I think that often others do.
> > > > > > > > >
> > > > > > > > > This seems to me to be a case that the benefit of removing
> > this
> > > > is
> > > > > > > > minimal
> > > > > > > > > and could cause quite a lot of disruption to users.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > <http://www.trimble.com/>
> > > > > > > > Raymond Wilson
> > > > > > > > Solution Architect, Civil Construction Software Systems
> (CCSS)
> > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > > +64-21-2013317 Mobile
> > > > > > > > raymond_wilson@trimble.com
> > > > > > > >
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Alexei Scherbakov
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > <http://www.trimble.com/>
> > > > > > Raymond Wilson
> > > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > +64-21-2013317 Mobile
> > > > > > raymond_wilson@trimble.com
> > > > > >
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Alexei Scherbakov
> > > > >
> > > >
> > >
> > >
> > > --
> > > <http://www.trimble.com/>
> > > Raymond Wilson
> > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > 11 Birmingham Drive | Christchurch, New Zealand
> > > +64-21-2013317 Mobile
> > > raymond_wilson@trimble.com
> > >
> > > <
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > >
> > >
> >
>
>
> --
> <http://www.trimble.com/>
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> +64-21-2013317 Mobile
> raymond_wilson@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>
Re: Custom Affinity Functions proposed for removal?
Posted by Raymond Wilson <ra...@trimble.com>.
If I have a primary key vector like this <x><y><t>, where <x> & <y> can
have varying values for <t>, and I want all keys with the same <x> and <y>
to reside in the same partition for processing colocation requirements then
I can't use the standard Ignite mapping to do this, I need to use a custom
mapper than uses just the <x> & <y> components.
On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Raymond,
>
> Thanks for the details. So it sounds like you have a custom affinity
> mapper, not affinity function. This makes things simpler, but I'm still
> failing to understand why standard mechanisms for collocation [1] didn't
> work for you. Could you please clarify?
>
> [1]
> https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
>
> -Val
>
> On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <raymond_wilson@trimble.com
> >
> wrote:
>
> > In terms of Key -> Partition -> Node mapping, we provide customer
> affinity
> > mappers for the Key -> Partition stage an allow Ignite to map partitions
> to
> > nodes.
> >
> > Our keys are structs with multiple fields forming a composite primary
> key,
> > parts of which are spatially identifying and parts contain other
> > characteristics. We want to ensure all related spatial elements resolve
> to
> > the same partition and so will be placed on the same node for query
> > colocation purposes.
> >
> >
> >
> > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Moti, Raymond,
> > >
> > > Could you please describe your use cases in more detail? What are the
> > types
> > > used as cache keys? What is the custom logic that you use for affinity
> > > mapping? What was the exact reason to customize versus using built-in
> > > collocation mechanisms?
> > >
> > > Ultimately, I'm sure that custom affinity functions will go away cause
> > they
> > > introduce multiple potential issues. However, I would definitely like
> to
> > > make sure that your use cases are still supported in Ignite 3.0 via
> some
> > > other means.
> > >
> > > -Val
> > >
> > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > > alexey.scherbakoff@gmail.com> wrote:
> > >
> > > > Thanks for the clarification.
> > > >
> > > > There was no intention to remove the customizable key to partition
> > > mapping.
> > > >
> > > > Difficulties arise when mapping partitions to nodes, so it's
> desirable
> > to
> > > > have internally tested implementation with a way to customize it's
> > > behavior
> > > > without additional coding on the user side.
> > > >
> > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> > raymond_wilson@trimble.com
> > > >:
> > > >
> > > > > Just to be clear, the affinity functions we are using convert keys
> to
> > > > > partitions, we do not map partitions to nodes and leave that to
> > Ignite.
> > > > >
> > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > >
> > > > > > Hello.
> > > > > >
> > > > > > Custom affinity functions can cause weird bugs and data loss if
> > > > > implemented
> > > > > > wrongly.
> > > > > > There is an intention to keep a backup filter based on user
> > > attributes
> > > > > > (with additional validation logic to ensure correctness) for
> > > > controllable
> > > > > > data placement.
> > > > > >
> > > > > > Can you describe more precisely why you had to implement custom
> > > > affinity
> > > > > > functions and not resort to default rendezvous affinity + backup
> > > > filter ?
> > > > > >
> > > > > >
> > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > > > raymond_wilson@trimble.com
> > > > > >:
> > > > > >
> > > > > > > We also use custom affinity functions (vis the C# client).
> > > > > > >
> > > > > > > The wish list mentions use of a particular annotation
> > > > > > > (@CentralizedAffinityFunction):
> > > > > > > Is the wish to remove just this annotation, or the ability to
> > > define
> > > > > > custom
> > > > > > > affinity functions at all?
> > > > > > >
> > > > > > > In our case we use affinity functions to ensure particular
> > > > distribution
> > > > > > of
> > > > > > > spatial data across a processing cluster to ensure even load
> > > > > management.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > > > moti.nisenson@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I saw at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > > that custom affinity functions are on the potential wishlist
> > for
> > > > > > removal.
> > > > > > > > The way we're using it's very critical that we be able to
> > control
> > > > the
> > > > > > > > placement of data quite precisely - as part of that we
> specify
> > > > > > explicitly
> > > > > > > > the partition we want in the key, and then our affinity
> > function
> > > > uses
> > > > > > > that
> > > > > > > > (else delegating to default rendezvous). We don't need all
> the
> > > > > > > > abilities there, although I think that often others do.
> > > > > > > >
> > > > > > > > This seems to me to be a case that the benefit of removing
> this
> > > is
> > > > > > > minimal
> > > > > > > > and could cause quite a lot of disruption to users.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > <http://www.trimble.com/>
> > > > > > > Raymond Wilson
> > > > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > +64-21-2013317 Mobile
> > > > > > > raymond_wilson@trimble.com
> > > > > > >
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best regards,
> > > > > > Alexei Scherbakov
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > <http://www.trimble.com/>
> > > > > Raymond Wilson
> > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > +64-21-2013317 Mobile
> > > > > raymond_wilson@trimble.com
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> >
> >
> > --
> > <http://www.trimble.com/>
> > Raymond Wilson
> > Solution Architect, Civil Construction Software Systems (CCSS)
> > 11 Birmingham Drive | Christchurch, New Zealand
> > +64-21-2013317 Mobile
> > raymond_wilson@trimble.com
> >
> > <
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > >
> >
>
--
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com
<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
Re: Custom Affinity Functions proposed for removal?
Posted by Valentin Kulichenko <va...@gmail.com>.
Raymond,
Thanks for the details. So it sounds like you have a custom affinity
mapper, not affinity function. This makes things simpler, but I'm still
failing to understand why standard mechanisms for collocation [1] didn't
work for you. Could you please clarify?
[1] https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
-Val
On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <ra...@trimble.com>
wrote:
> In terms of Key -> Partition -> Node mapping, we provide customer affinity
> mappers for the Key -> Partition stage an allow Ignite to map partitions to
> nodes.
>
> Our keys are structs with multiple fields forming a composite primary key,
> parts of which are spatially identifying and parts contain other
> characteristics. We want to ensure all related spatial elements resolve to
> the same partition and so will be placed on the same node for query
> colocation purposes.
>
>
>
> On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Moti, Raymond,
> >
> > Could you please describe your use cases in more detail? What are the
> types
> > used as cache keys? What is the custom logic that you use for affinity
> > mapping? What was the exact reason to customize versus using built-in
> > collocation mechanisms?
> >
> > Ultimately, I'm sure that custom affinity functions will go away cause
> they
> > introduce multiple potential issues. However, I would definitely like to
> > make sure that your use cases are still supported in Ignite 3.0 via some
> > other means.
> >
> > -Val
> >
> > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com> wrote:
> >
> > > Thanks for the clarification.
> > >
> > > There was no intention to remove the customizable key to partition
> > mapping.
> > >
> > > Difficulties arise when mapping partitions to nodes, so it's desirable
> to
> > > have internally tested implementation with a way to customize it's
> > behavior
> > > without additional coding on the user side.
> > >
> > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> raymond_wilson@trimble.com
> > >:
> > >
> > > > Just to be clear, the affinity functions we are using convert keys to
> > > > partitions, we do not map partitions to nodes and leave that to
> Ignite.
> > > >
> > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > alexey.scherbakoff@gmail.com> wrote:
> > > >
> > > > > Hello.
> > > > >
> > > > > Custom affinity functions can cause weird bugs and data loss if
> > > > implemented
> > > > > wrongly.
> > > > > There is an intention to keep a backup filter based on user
> > attributes
> > > > > (with additional validation logic to ensure correctness) for
> > > controllable
> > > > > data placement.
> > > > >
> > > > > Can you describe more precisely why you had to implement custom
> > > affinity
> > > > > functions and not resort to default rendezvous affinity + backup
> > > filter ?
> > > > >
> > > > >
> > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > > raymond_wilson@trimble.com
> > > > >:
> > > > >
> > > > > > We also use custom affinity functions (vis the C# client).
> > > > > >
> > > > > > The wish list mentions use of a particular annotation
> > > > > > (@CentralizedAffinityFunction):
> > > > > > Is the wish to remove just this annotation, or the ability to
> > define
> > > > > custom
> > > > > > affinity functions at all?
> > > > > >
> > > > > > In our case we use affinity functions to ensure particular
> > > distribution
> > > > > of
> > > > > > spatial data across a processing cluster to ensure even load
> > > > management.
> > > > > >
> > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > > moti.nisenson@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I saw at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > that custom affinity functions are on the potential wishlist
> for
> > > > > removal.
> > > > > > > The way we're using it's very critical that we be able to
> control
> > > the
> > > > > > > placement of data quite precisely - as part of that we specify
> > > > > explicitly
> > > > > > > the partition we want in the key, and then our affinity
> function
> > > uses
> > > > > > that
> > > > > > > (else delegating to default rendezvous). We don't need all the
> > > > > > > abilities there, although I think that often others do.
> > > > > > >
> > > > > > > This seems to me to be a case that the benefit of removing this
> > is
> > > > > > minimal
> > > > > > > and could cause quite a lot of disruption to users.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > <http://www.trimble.com/>
> > > > > > Raymond Wilson
> > > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > +64-21-2013317 Mobile
> > > > > > raymond_wilson@trimble.com
> > > > > >
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Alexei Scherbakov
> > > > >
> > > >
> > > >
> > > > --
> > > > <http://www.trimble.com/>
> > > > Raymond Wilson
> > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > +64-21-2013317 Mobile
> > > > raymond_wilson@trimble.com
> > > >
> > > > <
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
> --
> <http://www.trimble.com/>
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> +64-21-2013317 Mobile
> raymond_wilson@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>
Re: Custom Affinity Functions proposed for removal?
Posted by Raymond Wilson <ra...@trimble.com>.
In terms of Key -> Partition -> Node mapping, we provide customer affinity
mappers for the Key -> Partition stage an allow Ignite to map partitions to
nodes.
Our keys are structs with multiple fields forming a composite primary key,
parts of which are spatially identifying and parts contain other
characteristics. We want to ensure all related spatial elements resolve to
the same partition and so will be placed on the same node for query
colocation purposes.
On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Moti, Raymond,
>
> Could you please describe your use cases in more detail? What are the types
> used as cache keys? What is the custom logic that you use for affinity
> mapping? What was the exact reason to customize versus using built-in
> collocation mechanisms?
>
> Ultimately, I'm sure that custom affinity functions will go away cause they
> introduce multiple potential issues. However, I would definitely like to
> make sure that your use cases are still supported in Ignite 3.0 via some
> other means.
>
> -Val
>
> On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > Thanks for the clarification.
> >
> > There was no intention to remove the customizable key to partition
> mapping.
> >
> > Difficulties arise when mapping partitions to nodes, so it's desirable to
> > have internally tested implementation with a way to customize it's
> behavior
> > without additional coding on the user side.
> >
> > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <raymond_wilson@trimble.com
> >:
> >
> > > Just to be clear, the affinity functions we are using convert keys to
> > > partitions, we do not map partitions to nodes and leave that to Ignite.
> > >
> > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > alexey.scherbakoff@gmail.com> wrote:
> > >
> > > > Hello.
> > > >
> > > > Custom affinity functions can cause weird bugs and data loss if
> > > implemented
> > > > wrongly.
> > > > There is an intention to keep a backup filter based on user
> attributes
> > > > (with additional validation logic to ensure correctness) for
> > controllable
> > > > data placement.
> > > >
> > > > Can you describe more precisely why you had to implement custom
> > affinity
> > > > functions and not resort to default rendezvous affinity + backup
> > filter ?
> > > >
> > > >
> > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > raymond_wilson@trimble.com
> > > >:
> > > >
> > > > > We also use custom affinity functions (vis the C# client).
> > > > >
> > > > > The wish list mentions use of a particular annotation
> > > > > (@CentralizedAffinityFunction):
> > > > > Is the wish to remove just this annotation, or the ability to
> define
> > > > custom
> > > > > affinity functions at all?
> > > > >
> > > > > In our case we use affinity functions to ensure particular
> > distribution
> > > > of
> > > > > spatial data across a processing cluster to ensure even load
> > > management.
> > > > >
> > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > moti.nisenson@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I saw at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > that custom affinity functions are on the potential wishlist for
> > > > removal.
> > > > > > The way we're using it's very critical that we be able to control
> > the
> > > > > > placement of data quite precisely - as part of that we specify
> > > > explicitly
> > > > > > the partition we want in the key, and then our affinity function
> > uses
> > > > > that
> > > > > > (else delegating to default rendezvous). We don't need all the
> > > > > > abilities there, although I think that often others do.
> > > > > >
> > > > > > This seems to me to be a case that the benefit of removing this
> is
> > > > > minimal
> > > > > > and could cause quite a lot of disruption to users.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > <http://www.trimble.com/>
> > > > > Raymond Wilson
> > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > +64-21-2013317 Mobile
> > > > > raymond_wilson@trimble.com
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> > >
> > > --
> > > <http://www.trimble.com/>
> > > Raymond Wilson
> > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > 11 Birmingham Drive | Christchurch, New Zealand
> > > +64-21-2013317 Mobile
> > > raymond_wilson@trimble.com
> > >
> > > <
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>
--
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com
<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
Re: Custom Affinity Functions proposed for removal?
Posted by Valentin Kulichenko <va...@gmail.com>.
Moti, Raymond,
Could you please describe your use cases in more detail? What are the types
used as cache keys? What is the custom logic that you use for affinity
mapping? What was the exact reason to customize versus using built-in
collocation mechanisms?
Ultimately, I'm sure that custom affinity functions will go away cause they
introduce multiple potential issues. However, I would definitely like to
make sure that your use cases are still supported in Ignite 3.0 via some
other means.
-Val
On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:
> Thanks for the clarification.
>
> There was no intention to remove the customizable key to partition mapping.
>
> Difficulties arise when mapping partitions to nodes, so it's desirable to
> have internally tested implementation with a way to customize it's behavior
> without additional coding on the user side.
>
> пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <ra...@trimble.com>:
>
> > Just to be clear, the affinity functions we are using convert keys to
> > partitions, we do not map partitions to nodes and leave that to Ignite.
> >
> > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com> wrote:
> >
> > > Hello.
> > >
> > > Custom affinity functions can cause weird bugs and data loss if
> > implemented
> > > wrongly.
> > > There is an intention to keep a backup filter based on user attributes
> > > (with additional validation logic to ensure correctness) for
> controllable
> > > data placement.
> > >
> > > Can you describe more precisely why you had to implement custom
> affinity
> > > functions and not resort to default rendezvous affinity + backup
> filter ?
> > >
> > >
> > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> raymond_wilson@trimble.com
> > >:
> > >
> > > > We also use custom affinity functions (vis the C# client).
> > > >
> > > > The wish list mentions use of a particular annotation
> > > > (@CentralizedAffinityFunction):
> > > > Is the wish to remove just this annotation, or the ability to define
> > > custom
> > > > affinity functions at all?
> > > >
> > > > In our case we use affinity functions to ensure particular
> distribution
> > > of
> > > > spatial data across a processing cluster to ensure even load
> > management.
> > > >
> > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> moti.nisenson@gmail.com>
> > > > wrote:
> > > >
> > > > > I saw at
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > that custom affinity functions are on the potential wishlist for
> > > removal.
> > > > > The way we're using it's very critical that we be able to control
> the
> > > > > placement of data quite precisely - as part of that we specify
> > > explicitly
> > > > > the partition we want in the key, and then our affinity function
> uses
> > > > that
> > > > > (else delegating to default rendezvous). We don't need all the
> > > > > abilities there, although I think that often others do.
> > > > >
> > > > > This seems to me to be a case that the benefit of removing this is
> > > > minimal
> > > > > and could cause quite a lot of disruption to users.
> > > > >
> > > > > Thanks!
> > > > >
> > > >
> > > >
> > > > --
> > > > <http://www.trimble.com/>
> > > > Raymond Wilson
> > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > +64-21-2013317 Mobile
> > > > raymond_wilson@trimble.com
> > > >
> > > > <
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
> >
> > --
> > <http://www.trimble.com/>
> > Raymond Wilson
> > Solution Architect, Civil Construction Software Systems (CCSS)
> > 11 Birmingham Drive | Christchurch, New Zealand
> > +64-21-2013317 Mobile
> > raymond_wilson@trimble.com
> >
> > <
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
Re: Custom Affinity Functions proposed for removal?
Posted by Alexei Scherbakov <al...@gmail.com>.
Thanks for the clarification.
There was no intention to remove the customizable key to partition mapping.
Difficulties arise when mapping partitions to nodes, so it's desirable to
have internally tested implementation with a way to customize it's behavior
without additional coding on the user side.
пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <ra...@trimble.com>:
> Just to be clear, the affinity functions we are using convert keys to
> partitions, we do not map partitions to nodes and leave that to Ignite.
>
> On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > Hello.
> >
> > Custom affinity functions can cause weird bugs and data loss if
> implemented
> > wrongly.
> > There is an intention to keep a backup filter based on user attributes
> > (with additional validation logic to ensure correctness) for controllable
> > data placement.
> >
> > Can you describe more precisely why you had to implement custom affinity
> > functions and not resort to default rendezvous affinity + backup filter ?
> >
> >
> > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <raymond_wilson@trimble.com
> >:
> >
> > > We also use custom affinity functions (vis the C# client).
> > >
> > > The wish list mentions use of a particular annotation
> > > (@CentralizedAffinityFunction):
> > > Is the wish to remove just this annotation, or the ability to define
> > custom
> > > affinity functions at all?
> > >
> > > In our case we use affinity functions to ensure particular distribution
> > of
> > > spatial data across a processing cluster to ensure even load
> management.
> > >
> > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <mo...@gmail.com>
> > > wrote:
> > >
> > > > I saw at
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > that custom affinity functions are on the potential wishlist for
> > removal.
> > > > The way we're using it's very critical that we be able to control the
> > > > placement of data quite precisely - as part of that we specify
> > explicitly
> > > > the partition we want in the key, and then our affinity function uses
> > > that
> > > > (else delegating to default rendezvous). We don't need all the
> > > > abilities there, although I think that often others do.
> > > >
> > > > This seems to me to be a case that the benefit of removing this is
> > > minimal
> > > > and could cause quite a lot of disruption to users.
> > > >
> > > > Thanks!
> > > >
> > >
> > >
> > > --
> > > <http://www.trimble.com/>
> > > Raymond Wilson
> > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > 11 Birmingham Drive | Christchurch, New Zealand
> > > +64-21-2013317 Mobile
> > > raymond_wilson@trimble.com
> > >
> > > <
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>
>
> --
> <http://www.trimble.com/>
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> +64-21-2013317 Mobile
> raymond_wilson@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>
--
Best regards,
Alexei Scherbakov
Re: Custom Affinity Functions proposed for removal?
Posted by Raymond Wilson <ra...@trimble.com>.
Just to be clear, the affinity functions we are using convert keys to
partitions, we do not map partitions to nodes and leave that to Ignite.
On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:
> Hello.
>
> Custom affinity functions can cause weird bugs and data loss if implemented
> wrongly.
> There is an intention to keep a backup filter based on user attributes
> (with additional validation logic to ensure correctness) for controllable
> data placement.
>
> Can you describe more precisely why you had to implement custom affinity
> functions and not resort to default rendezvous affinity + backup filter ?
>
>
> пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <ra...@trimble.com>:
>
> > We also use custom affinity functions (vis the C# client).
> >
> > The wish list mentions use of a particular annotation
> > (@CentralizedAffinityFunction):
> > Is the wish to remove just this annotation, or the ability to define
> custom
> > affinity functions at all?
> >
> > In our case we use affinity functions to ensure particular distribution
> of
> > spatial data across a processing cluster to ensure even load management.
> >
> > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <mo...@gmail.com>
> > wrote:
> >
> > > I saw at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > that custom affinity functions are on the potential wishlist for
> removal.
> > > The way we're using it's very critical that we be able to control the
> > > placement of data quite precisely - as part of that we specify
> explicitly
> > > the partition we want in the key, and then our affinity function uses
> > that
> > > (else delegating to default rendezvous). We don't need all the
> > > abilities there, although I think that often others do.
> > >
> > > This seems to me to be a case that the benefit of removing this is
> > minimal
> > > and could cause quite a lot of disruption to users.
> > >
> > > Thanks!
> > >
> >
> >
> > --
> > <http://www.trimble.com/>
> > Raymond Wilson
> > Solution Architect, Civil Construction Software Systems (CCSS)
> > 11 Birmingham Drive | Christchurch, New Zealand
> > +64-21-2013317 Mobile
> > raymond_wilson@trimble.com
> >
> > <
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
--
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com
<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
Re: Custom Affinity Functions proposed for removal?
Posted by Alexei Scherbakov <al...@gmail.com>.
Hello.
Custom affinity functions can cause weird bugs and data loss if implemented
wrongly.
There is an intention to keep a backup filter based on user attributes
(with additional validation logic to ensure correctness) for controllable
data placement.
Can you describe more precisely why you had to implement custom affinity
functions and not resort to default rendezvous affinity + backup filter ?
пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <ra...@trimble.com>:
> We also use custom affinity functions (vis the C# client).
>
> The wish list mentions use of a particular annotation
> (@CentralizedAffinityFunction):
> Is the wish to remove just this annotation, or the ability to define custom
> affinity functions at all?
>
> In our case we use affinity functions to ensure particular distribution of
> spatial data across a processing cluster to ensure even load management.
>
> On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <mo...@gmail.com>
> wrote:
>
> > I saw at
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > that custom affinity functions are on the potential wishlist for removal.
> > The way we're using it's very critical that we be able to control the
> > placement of data quite precisely - as part of that we specify explicitly
> > the partition we want in the key, and then our affinity function uses
> that
> > (else delegating to default rendezvous). We don't need all the
> > abilities there, although I think that often others do.
> >
> > This seems to me to be a case that the benefit of removing this is
> minimal
> > and could cause quite a lot of disruption to users.
> >
> > Thanks!
> >
>
>
> --
> <http://www.trimble.com/>
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> +64-21-2013317 Mobile
> raymond_wilson@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>
--
Best regards,
Alexei Scherbakov
Re: Custom Affinity Functions proposed for removal?
Posted by Raymond Wilson <ra...@trimble.com>.
We also use custom affinity functions (vis the C# client).
The wish list mentions use of a particular annotation
(@CentralizedAffinityFunction):
Is the wish to remove just this annotation, or the ability to define custom
affinity functions at all?
In our case we use affinity functions to ensure particular distribution of
spatial data across a processing cluster to ensure even load management.
On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <mo...@gmail.com>
wrote:
> I saw at
>
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> that custom affinity functions are on the potential wishlist for removal.
> The way we're using it's very critical that we be able to control the
> placement of data quite precisely - as part of that we specify explicitly
> the partition we want in the key, and then our affinity function uses that
> (else delegating to default rendezvous). We don't need all the
> abilities there, although I think that often others do.
>
> This seems to me to be a case that the benefit of removing this is minimal
> and could cause quite a lot of disruption to users.
>
> Thanks!
>
--
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com
<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>