Re: AI 3.0: writeSynchronizationMode re-thinking

[Anton Vinogradov <av...@apache.org>]

Sergey,

Sorry for the late response.

I'd think twice or more :) and discussed the idea with Alexey G.
Seems, we have no reason to check some backups since AI should guarantee it
has everything consistent.
But, it useful to have some feature to check this guarantee is not bluff.

BTW, I'm still ok with your proposal.

On Tue, Apr 30, 2019 at 9:24 PM Sergey Kozlov <sk...@gridgain.com> wrote:

> Anton
>
> I'm Ok with your proposal but IMO  it should be provided as IEP?
>
> On Mon, Apr 29, 2019 at 4:05 PM Anton Vinogradov <av...@apache.org> wrote:
>
> > Sergey,
> >
> > I'd like to continue the discussion since it closely linked to the
> problem
> > I'm currently working on.
> >
> > 1) writeSynchronizationMode should not be a part of cache configuration,
> > agree.
> > This should be up to the user to decide how strong should be "update
> > guarantee".
> > So, I propose to have a special cache proxy, .withBlaBla() (at 3.x).
> >
> > 2) Primary fail on !FULL_SYNC is not the single problem leads to an
> > inconsistent state.
> > Bugs and incorrect recovery also cause the same problem.
> >
> > Currently, we have a solution [1] to check cluster to be consistent, but
> it
> > has a bad resolution (will tell you only what partitions are broken).
> > So, to find the broken entries you need some special API, which will
> check
> > all copies and let you know what's went wrong.
> >
> > 3) Since we mostly agree that write should affect some backups in sync
> way,
> > how about to have similar logic for reading?
> >
> > So, I propose to have special proxy .withQuorumRead(backupsCnt) which
> will
> > check the explicit amount of backups on each read and return you the
> latest
> > values.
> > This proxy already implemented [2] for all copies, but I'm going to
> extend
> > it with explicit backups number.
> >
> > Thoughts?
> >
> > 3.1) Backups can be checked in two ways:
> > - request data from all backups, but wait for explicit number (solves the
> > slow backup issue, but produce traffic)
> > - request data from an explicit number of backups (less traffic, but can
> be
> > as slow as all copies check case)
> > what strategy is better? Should it be configurable?
> >
> > [1]
> >
> >
> https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums
> > [2] https://issues.apache.org/jira/browse/IGNITE-10663
> >
> > On Thu, Apr 25, 2019 at 7:04 PM Sergey Kozlov <sk...@gridgain.com>
> > wrote:
> >
> > > There's another point to improve:
> > > if  *syncPartitions=N* comes as the configurable in run-time it will
> > allow
> > > to manage the consistency-performance balance runtime, e.g. switch to
> > full
> > > async for preloading and then go to back to full sync for regular
> > > operations
> > >
> > >
> > > On Thu, Apr 25, 2019 at 6:48 PM Sergey Kozlov <sk...@gridgain.com>
> > > wrote:
> > >
> > > > Vyacheskav,
> > > >
> > > > You're right with the referring to MongoDB doc. In general the idea
> is
> > > > very similar. Many vendors use such approach (1).
> > > >
> > > > [1]
> > > >
> > >
> >
> https://dev.mysql.com/doc/refman/8.0/en/replication-options-master.html#sysvar_rpl_semi_sync_master_wait_for_slave_count
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Apr 25, 2019 at 6:40 PM Vyacheslav Daradur <
> > daradurvs@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi, Sergey,
> > > >>
> > > >> Makes sense to me in case of performance issues, but may lead to
> > losing
> > > >> data.
> > > >>
> > > >> >> *by the new option *syncPartitions=N* (not best name just for
> > > >> referring)
> > > >>
> > > >> Seems similar to "Write Concern"[1] in MongoDB. It is used in the
> same
> > > >> way as you described.
> > > >>
> > > >> On the other hand, if you have such issues it should be investigated
> > > >> first: why it causes performance drops: network issues etc.
> > > >>
> > > >> [1] https://docs.mongodb.com/manual/reference/write-concern/
> > > >>
> > > >> On Thu, Apr 25, 2019 at 6:24 PM Sergey Kozlov <skozlov@gridgain.com
> >
> > > >> wrote:
> > > >> >
> > > >> > Ilya
> > > >> >
> > > >> > See comments inline.
> > > >> > On Thu, Apr 25, 2019 at 5:11 PM Ilya Kasnacheev <
> > > >> ilya.kasnacheev@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hello!
> > > >> > >
> > > >> > > When you have 2 backups and N = 1, how will conflicts be
> resolved?
> > > >> > >
> > > >> >
> > > >> > > Imagine that you had N = 1, and primary node failed immediately
> > > after
> > > >> > > operation. Now you have one backup that was updated
> synchronously
> > > and
> > > >> one
> > > >> > > which did not. Will they stay unsynced, or is there any
> mechanism
> > of
> > > >> > > re-syncing?
> > > >> > >
> > > >> >
> > > >> > Same way as Ignite processes the failures for PRIMARY_SYNC.
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > Why would one want to "update for 1 primary and 1 backup
> > > >> synchronously,
> > > >> > > update the rest of backup partitions asynchronously"? What's the
> > use
> > > >> case?
> > > >> > >
> > > >> >
> > > >> > The case to have more backups but do not pay the performance
> penalty
> > > for
> > > >> > that :)
> > > >> > For the distributed systems one backup looks like risky. But more
> > > >> backups
> > > >> > directly impacts to performance.
> > > >> > Other point is to split the strict consistent apps like bank apps
> > and
> > > >> the
> > > >> > other apps like fraud detection, analytics, reports and so on.
> > > >> > In that case you can configure partitions distribution by a custom
> > > >> affinity
> > > >> > and have following:
> > > >> >  - first set of nodes for critical (from consistency point)
> > operations
> > > >> >  - second set of nodes have async backup partitions only for other
> > > >> > operations (reports, analytics)
> > > >> >
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > Regards,
> > > >> > > --
> > > >> > > Ilya Kasnacheev
> > > >> > >
> > > >> > >
> > > >> > > чт, 25 апр. 2019 г. в 16:55, Sergey Kozlov <
> skozlov@gridgain.com
> > >:
> > > >> > >
> > > >> > > > Igniters
> > > >> > > >
> > > >> > > > I'm working with the wide range of cache configurations and
> > found
> > > >> (from
> > > >> > > my
> > > >> > > > standpoint) the interesting point for the discussion:
> > > >> > > >
> > > >> > > > Now we have following *writeSynchronizationMode *options:
> > > >> > > >
> > > >> > > >    1. *FULL_ASYNC*
> > > >> > > >       -  primary partition updated asynchronously
> > > >> > > >       -  backup partitions updated asynchronously
> > > >> > > >    2. *PRIMARY_SYNC*
> > > >> > > >       - primary partition updated synchronously
> > > >> > > >       - backup partitions updated asynchronously
> > > >> > > >    3. *FULL_SYNC*
> > > >> > > >       - primary partition updated synchronously
> > > >> > > >       - backup partitions updated synchronously
> > > >> > > >
> > > >> > > > The approach above is covering everything if you've 0 or 1
> > backup.
> > > >> > > > But for 2 or more backups we can't reach the following case
> > > >> (something
> > > >> > > > between *PRIMARY_SYNC *and *FULL_SYNC*):
> > > >> > > >  - update for 1 primary and 1 backup synchronously
> > > >> > > >  - update the rest of backup partitions asynchronously
> > > >> > > >
> > > >> > > > The idea is to join all current modes into single one and
> > replace
> > > >> > > > *writeSynchronizationMode
> > > >> > > > *by the new option *syncPartitions=N* (not best name just for
> > > >> referring)
> > > >> > > > covers the approach:
> > > >> > > >
> > > >> > > >    - N = 0 means *FULL_ASYNC*
> > > >> > > >    - N = (backups+1) means *FULL_SYNC*
> > > >> > > >    - 0 < N < (backups+1) means either *PRIMARY_SYNC *(N=1) or
> > new
> > > >> mode
> > > >> > > >    described above
> > > >> > > >
> > > >> > > > IMO it will allow to make more flexible and consistent
> > > >> configurations
> > > >> > > >
> > > >> > > > --
> > > >> > > > Sergey Kozlov
> > > >> > > > GridGain Systems
> > > >> > > > www.gridgain.com
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Sergey Kozlov
> > > >> > GridGain Systems
> > > >> > www.gridgain.com
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best Regards, Vyacheslav D.
> > > >>
> > > >
> > > >
> > > > --
> > > > Sergey Kozlov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> > >
> > > --
> > > Sergey Kozlov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>