Node hangs while accessing cache from Compute Task

[Gaurav Bajaj <ga...@gmail.com>]

Hi,



I am facing issue when I try to access cache from compute task which is
triggered from Continuous Query listener.

This issue happens only when listener is called back synchronously (i.e
when @IgniteAsyncCallback is NOT used).

Cluster node simply hangs and no error etc is thrown.

I have attached the sample application source code to reproduce this issue.



 Sample application details :



1.  App.Java is entry point of the application. Please change the path of
default-config.xml and start the application.

2. Two Caches Employee and Project Cache holding Employee and Project
object.

3. Employee holds key (ProjectId) of Person object.

4. Person holds field called "status"

5. Add continuous query which will be triggered on changing value of
"status" field in Project.

6. Listener will trigger compute task (Map-Reduce Job)

7. In Compute Job, get all the employees of this Project. In Map step,
simply return employee salary

8. In reduce, aggregate the salaries of all the employees.

9. If in the Map job, we try to access Project Cache then Node hangs.

10.If the Listener in Step 5 is annotated with @IgniteAsyncCallback, it
works without getting hanged.



Can you please check if am I doing something wrong or it's Ignite issue ?

According to me, Ignite should atleast throw exception.

Re: Node hangs while accessing cache from Compute Task

[GB <ga...@gmail.com>]

Thanks Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Node-hangs-while-accessing-cache-from-Compute-Task-tp10637p10790.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Node hangs while accessing cache from Compute Task

[dkarachentsev <dk...@gridgain.com>]

I haven't found such documentation, so opened a ticket to improve it [1].

[1] https://issues.apache.org/jira/browse/IGNITE-4718

Thanks!

-Dmitry



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Node-hangs-while-accessing-cache-from-Compute-Task-tp10637p10700.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Node hangs while accessing cache from Compute Task

[Gaurav Bajaj <ga...@gmail.com>]

Hi Dmitry,

Thanks for the detailed explanation.

Do you know if this behavior "It's not allowed to use cache API in
CacheEntryProcessor, query listeners, interceptors and so forth, because it
leads to deadlocks" is documented somewhere?


Thanks,
Gaurav


On Thu, Feb 16, 2017 at 12:16 PM, dkarachentsev <dk...@gridgain.com>
wrote:

> Hi,
>
> This is a correct behavior. It's not allowed to use cache API in
> CacheEntryProcessor, query listeners, interceptors and so forth, because it
> leads to deadlocks. In your example it looks like (threads T1, T2):
> T1 put entry -> T1 lock entry -> T1 call listener -> T1 execute job -> T1
> wait for result -> T2 get entry -> T2 try to lock entry -> T2 BLOCKED by T1
>
> @IgniteAsyncCallback is a legal way to resolve such situations. In your
> case
> it will be:
> T1 put entry -> T1 lock entry -> T1 submit listener to callback pool -> T1
> unlock entry.
> T3 - callback thread.
> T3 execute job -> T3 wait for result -> T2 get entry -> T2 lock entry -> T2
> unlock entry -> T2 return compute result -> T3 release with result.
>
> No deadlocks now.
>
> -Dmitry.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Node-hangs-while-accessing-cache-from-Compute-Task-
> tp10637p10668.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Node hangs while accessing cache from Compute Task

[dkarachentsev <dk...@gridgain.com>]

Hi,

This is a correct behavior. It's not allowed to use cache API in
CacheEntryProcessor, query listeners, interceptors and so forth, because it
leads to deadlocks. In your example it looks like (threads T1, T2):
T1 put entry -> T1 lock entry -> T1 call listener -> T1 execute job -> T1
wait for result -> T2 get entry -> T2 try to lock entry -> T2 BLOCKED by T1

@IgniteAsyncCallback is a legal way to resolve such situations. In your case
it will be:
T1 put entry -> T1 lock entry -> T1 submit listener to callback pool -> T1
unlock entry.
T3 - callback thread.
T3 execute job -> T3 wait for result -> T2 get entry -> T2 lock entry -> T2
unlock entry -> T2 return compute result -> T3 release with result.

No deadlocks now.

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Node-hangs-while-accessing-cache-from-Compute-Task-tp10637p10668.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.