Unintended merging of multiple transactions into a single transaction

["kimec.ethome.sk" <ki...@ethome.sk>]

Greetings,

while testing our Apache Ignite based app with Gatling, I have noticed 
that sometimes a write to a single cache fails, because it has different 
writeBehind setting than some other cache. The problem is that the two 
caches are not part of the same transaction and should have been updated 
in two different transactions, not one.

I do not have a reproducer at hand but the scenario is simple:
* Have server node bundled with your application
* Have at least two nodes
* Have two caches with PARTITIONED and TRANSACTIONAL
* Have one of the caches with writeBehind=true and the other one with 
writeBehind=false
* Use optimistic transaction with the first cache (SERIALIZED + 
OPTIMISTIC)
* Update both caches in sequence with two separate transactions but 
demarcate the first transaction with commitAsync() instead of commit()
* Have non-Ignite fixed size thread pool initiating the commits
* Shoot your application down with Gatling with no pauses or delays to 
stress the whole system

After some time you may get "Failed to enlist new cache to existing 
transaction" errors for operations that should not have been committed 
as a single transaction since the transaction boundary was demarcated 
with commitAsync().

I've tried to narrow down the issue to these sections:

https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxStateImpl.java#L266-L271

https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTransactionalCacheAdapter.java#L1112

My guess is that IgniteTxStateImpl is somehow thread local/or cached and 
under heavy load in a fixed sized thread pool, separate transactions are 
merged into one "batch" for optimization. The trouble is that 
commitAsync() in this case does not demarcate your transactions as you 
would have guessed.

Is this the expected behavior?

-- 
S pozdravom,

Kamil Mišúth

Re: Unintended merging of multiple transactions into a single transaction

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

I think that commitAsync() is not enough, you need to close() transaction
explicitly before starting a new one. Just a hunch.

Regards,
-- 
Ilya Kasnacheev


ср, 5 июн. 2019 г. в 10:19, kimec.ethome.sk <ki...@ethome.sk>:

> Greetings,
>
> while testing our Apache Ignite based app with Gatling, I have noticed
> that sometimes a write to a single cache fails, because it has different
> writeBehind setting than some other cache. The problem is that the two
> caches are not part of the same transaction and should have been updated
> in two different transactions, not one.
>
> I do not have a reproducer at hand but the scenario is simple:
> * Have server node bundled with your application
> * Have at least two nodes
> * Have two caches with PARTITIONED and TRANSACTIONAL
> * Have one of the caches with writeBehind=true and the other one with
> writeBehind=false
> * Use optimistic transaction with the first cache (SERIALIZED +
> OPTIMISTIC)
> * Update both caches in sequence with two separate transactions but
> demarcate the first transaction with commitAsync() instead of commit()
> * Have non-Ignite fixed size thread pool initiating the commits
> * Shoot your application down with Gatling with no pauses or delays to
> stress the whole system
>
> After some time you may get "Failed to enlist new cache to existing
> transaction" errors for operations that should not have been committed
> as a single transaction since the transaction boundary was demarcated
> with commitAsync().
>
> I've tried to narrow down the issue to these sections:
>
>
> https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxStateImpl.java#L266-L271
>
>
> https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTransactionalCacheAdapter.java#L1112
>
> My guess is that IgniteTxStateImpl is somehow thread local/or cached and
> under heavy load in a fixed sized thread pool, separate transactions are
> merged into one "batch" for optimization. The trouble is that
> commitAsync() in this case does not demarcate your transactions as you
> would have guessed.
>
> Is this the expected behavior?
>
> --
> S pozdravom,
>
> Kamil Mišúth
>