You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by bintisepaha <bi...@tudor.com> on 2016/07/13 01:37:37 UTC
Re: transaction not timing out
If in any case commit() is unable to finish, how can we catch that? in an
optimistic transaction, it should not even reach the commit phase using the
2 phase protocol.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6241.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by bintisepaha <bi...@tudor.com>.
Update - we started using pessimistic txns now and it looks much better. so
we can close this thread now.
raised another issue here
http://apache-ignite-users.70518.x6.nabble.com/IgniteCompute-broadcast-stuck-td7255.html
Thanks,
Binti
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p7256.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by bintisepaha <bi...@tudor.com>.
Val, we are still seeing this issue from time to time.
Transactions are hung during commit() . we do not use write-through so DB
will not cause slowness for us. I understand that prior to commit, locks
were acquired in prepare phase, so if prepare is successful, then why would
commit hang? I am assuming within the commit() call, the first phase of
prepare won't even complete if locks were not acquired, in that case we have
seen transaction optimistic exception various times and on retry the txn
completes successfully.
in the case where it hangs on commit(), we are unable to figure out what
resource it is blocked on?
Also, to free up the resource and see how the cluster behaves, if we kill
any of the offending nodes then the entire cluster hangs.
we use full_sync mode for caches, is there any setting that we need to use
with that? most of our distributed caches have a config similar to below
Any help is very much appreciated.
<bean id="tradeOrderCache"
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="TradeOrder" />
<property name="readThrough" value="True" />
<property name="writeThrough" value="True" />
<property name="writeBehindEnabled" value="True" />
<property name="writeBehindFlushFrequency" value="#{5 * 1000}" />
<property name="writeBehindFlushSize" value="0" />
<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="backups" value="1" />
<property name="startSize" value="#{1 * 1024 * 1024}" />
<property name="rebalanceBatchSize" value="#{2 * 1024 * 1024}"/>
<property name="rebalanceThrottle" value="100"/>
<property name="memoryMode" value="OFFHEAP_TIERED" />
<property name="offHeapMaxMemory" value="#{10 * 1024L * 1024L * 1024L}" />
<property name="cacheStoreFactory">
<bean class="javax.cache.configuration.FactoryBuilder$SingletonFactory">
<constructor-arg>
<ref bean="tradeOrderCacheStore" />
</constructor-arg>
</bean>
</property>
<property name="writeSynchronizationMode" value="FULL_SYNC" />
<property name="statisticsEnabled" value="true" />
<property name="queryEntities">
<list>
<bean id="tradeOrderQueryEntity"
class="org.apache.ignite.cache.QueryEntity">
<property name="keyType"
value="com.tudor.datagridI.client.data.trading.OrderKey" />
<property name="valueType"
value="com.tudor.datagridI.client.data.trading.TradeOrder" />
<property name="fields">
<map>
<entry key="traderId" value="java.lang.Integer" />
<entry key="orderId" value="java.lang.Integer" />
<entry key="insIid" value="java.lang.Integer" />
<entry key="settlement" value="java.util.Date" />
<entry key="clearAgent" value="java.lang.String" />
<entry key="strategy" value="java.lang.String" />
<entry key="pvId" value="java.lang.Integer" />
<entry key="pvDate" value="java.util.Date" />
<entry key="linkId" value="java.lang.Integer" />
</map>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<constructor-arg>
<list>
<value>traderId</value>
<value>orderId</value>
</list>
</constructor-arg>
<constructor-arg>
<value>SORTED</value>
</constructor-arg>
<property name="name" value="tradeOrder_key_index" />
</bean>
</list>
</property>
</bean>
</list>
</property>
<property name="affinity">
<bean
class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="excludeNeighbors" value="true" />
</bean>
</property>
</bean>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p7187.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by vkulichenko <va...@gmail.com>.
Yes, prepare phase happens within the commit() and the commit phase start
only after prepare is finished.
The commit phase updates the data which includes write-through to the
database. So if commit takes a lot of time due to issues with the database,
we can't rollback without causing data inconsistency.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6685.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by bintisepaha <bi...@tudor.com>.
Does prepare happen as part of commit() call?
As a user we only call commit(). so the txn should get to commit only if
prepare succeeded - right?
once in prepare mode, locks are acquired, no other txn should be able to
update these entries. then the txn goes to goes to commit() once locks are
acquired - what is holding it from updating the entries?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6638.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by vkulichenko <va...@gmail.com>.
Binti,
Locks are acquired on prepare phase, not on commit phase. In case of
optimistic transaction both phases happen one after another in a single
'tx.commit()' call.
Prepare phase is not a problem, because it's easy to detect collisions and
throw optimistic exception when all locks are not acquired yet. But on the
commit phase you actually update the data, so, as was mentioned before,
there is no reliable way to rollback. Basically, if you update one node,
then throw exception on the second and release locks, you get inconsistent
data.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6579.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by bintisepaha <bi...@tudor.com>.
What I mean is why would commit() hang? we use write behind, so cache store
updates are asynchronous (I doubt it would be an issue in our case)
during the commit() phase, if locks were not acquired on txn entries, we
would receive txn optimistic exception, we do receive that sometimes, but in
some other cases, what could be the possible reason for commit() to hang?
thread dump below for the hanging thread
Name: pub-#2%DataGridServer-Staging%
State: WAITING on
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture@6120e8e8
Total blocked: 11 Total waited: 584
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:155)
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:261)
com.tudor.datagridI.server.tradegen.OrderHolderSaveRunnable.run(OrderHolderSaveRunnable.java:268)
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1879)
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6397)
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1166)
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1770)
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
java.lang.Thread.run(Thread.java)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6576.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by vkulichenko <va...@gmail.com>.
Hi Binti,
bintisepaha wrote
> I do not understand why commit() is unable to finish. In an
> optimistic/serializable transaction, it should not even reach the commit
> phase using the 2 phase protocol. If the txn reached he commit() phase and
> then it was successful in acquiring all the locks it needs and it should
> not stall.
I'm not sure why the hang itself happens in your case but can you clarify
what you mean by "it should not even reach the commit phase using the 2
phase protocol"? Commit phase is the second one, that's when actual updates
happen, including CacheStore updates. You will very likely reach this phase
and any exceptions after can lead to inconsistent data, that's why Ignite
avoids them.
bintisepaha wrote
> Another related question, is it not recommended to start an ignite
> transaction from the client? what happens if the client crashes in the
> middle of the transaction? With pessimistic txn on client side, and if I
> kill the client in the middle of the txn, those keys that it had locks on
> are never released. now that same entry n cache cannot be updated by any
> process. how do I get out of such a situation.
This is the most common use case. If a node (can be either client or server,
doesn't matter) that executes the transaction fails, all locks acquired by
this transaction are released. If you observe different behavior, please
provide the reproducer.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6574.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: transaction not timing out
Posted by bintisepaha <bi...@tudor.com>.
Hi,
Trying to find an answer to the above.
I do not understand why commit() is unable to finish. In an
optimistic/serializable transaction, it should not even reach the commit
phase using the 2 phase protocol. If the txn reached he commit() phase and
then it was successful in acquiring all the locks it needs and it should not
stall.
Another related question, is it not recommended to start an ignite
transaction from the client? what happens if the client crashes in the
middle of the transaction? With pessimistic txn on client side, and if I
kill the client in the middle of the txn, those keys that it had locks on
are never released. now that same entry n cache cannot be updated by any
process. how do I get out of such a situation.
However, in optimistic locking this does not happen until commit, so it
behaves differently. But what if the client crashes during commit phase? i
also cannot use optimistic in this scenario.
Thanks,
Binti
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/transaction-not-timing-out-tp5540p6557.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.