Two-phase commit failure recovery

[akaptsan <ak...@mail.ru>]

I have read the block
http://gridgain.blogspot.co.uk/2014/09/two-phase-commit-for-in-memory-caches.html.
It is great! But I still have questions.

/
Whenever we cannot tell whether the database commit had happened or not, we
can simply reload the relevant data from database into cache upon committing
the transaction.  
/
1. Can Ignate reload the relevant data automaticaly or we must take care of
it? 
2. How can we detect this situation? 
3. Should we reload the whole cache or we can somehow see which objects must
be removed/reloaded?




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[vkulichenko <va...@gmail.com>]

Yes, CacheStore.load() will be called automatically in case coordinator node
fails in the middle of commit. But in your case I'm not quite sure it will
always work, because DB updates are actually asynchronous. Real synchronous
write-through to the DB is the only way to guarantee transactional
consistency between cache and DB.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9105.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[akaptsan <ak...@mail.ru>]

Can I be sure that in case of commit failure Ignite will always invoke
CacheStore.load() for all invalidated objects?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9103.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[akaptsan <ak...@mail.ru>]

We do use CacheStore write-through methods. Data are flushed to Kafka which
is used as intermediate persistent queue; data from the Kafka is written
into relational database asyncroniously. So from Ignite perspective it looks
like write-through, but in fact this is fault-tolerant write-behind.
  
We don't use read-through methods of CacheStore. Cache is pre-loaded by
means of IgniteDataStreamer. All CacheStore loading method are empty. 
We can't use CacheStore loading method because not all data are in database,
some of them can be still in Kafka. And there is no way to find an object
within Kafka queue.

That's why if commit failed Ignite will not be able to automatically ensure
consistency. We need to be able to detect such situation and repair cache
Repairing procedure could be:
- switch off cache usage
- reload cache
- switch on cache usage

Another variant: 
- suspend access to invalid objects
- flush Kafka to database (it can take several seconds)
- reload invalidated object from database
 





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9101.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[vkulichenko <va...@gmail.com>]

This blog is not about consistency within the cache, but about keeping cache
and the database consistent with each other. I.e., you can't end up with an
update that was successful in cache, but not successful in the database. And
automatic reloading is used to guarantee that in case of coordinator
failure.

If you don't have write-through (which can't be enabled without the
CacheStore), then the described case is not applicable. The consistency on
the cache itself is guaranteed without reloading step, see the first part of
the blog:
http://gridgain.blogspot.com/2014/09/two-phase-commit-for-distributed-in.html

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9099.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[akaptsan <ak...@mail.ru>]

We use SQL queries that's why we have to pre-load all data. For data loading
we use IgniteDataStreamer and don't use CacheStore. In case of two-phase
commit failure Ignite will not be able to automatically reload invalidated
data   



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9096.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[vkulichenko <va...@gmail.com>]

To be honest I'm not sure I understand what you mean because these two topics
are completely unrelated to each other. Can you elaborate?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9090.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

[akaptsan <ak...@mail.ru>]

Problem is we can't use standart cache loading due to
http://apache-ignite-users.70518.x6.nabble.com/SQL-query-against-cache-with-read-through-CacheStore-td9039.html
We need implement some custom logic to handle this situation: report the
problem to administrator, switch-off cache usage, invalidate cache(-s),
reload caches, switch-on cache usage ...




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9068.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Two-phase commit failure recovery

["alexey.goncharuk" <al...@gmail.com>]

Hi,

Note that the step described in the blog happens when a node that originated
a transaction crashes. In this case:
1. Ignite will reload the relevant data automatically
2. Not sure what you mean here. From a cluster point of view this happens
when a node leaves grid and a corresponding part of recovery protocol is
triggered. There is no need to do it manually
3. At the start of prepare step all the keys participating in the
transaction are known, so Ignite will reload only participating keys

Hope this helps,
AG



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.