You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Prasad Bhalerao <pr...@gmail.com> on 2018/06/06 17:30:06 UTC
Using Java executor service inside Ignite transaction
Hi Andrey, I am starting new mail thread as my previous mails were not
appearing in ignite archive even after sending it multiple times. But I am
receiving latest group mails.
*Question:* Is it safe to use Java executor service inside ignite
transaction to do parallel processing on same cache?
My affinity key is subscriptionId and one subscription Id can have multiple
asset groups.
In my application I submit the ignite runnable task using subscriptionId
affinity key. Now inside this ignite task I submit Callable tasks per
assetGroupId to java executor service to process (remove/add and update)
cache entries in parallel fashion
This whole thing is happening inside ignite transaction and it works
perfectly fine.
Problem occurs only when I update and then read the same entry and update
it again in same transaction. This is one particular corner case.
Andrey said "*In your case, task is executed in separate thread of executor
service which knows no nothing about transaction context and starts a new
transaction in new context.*
*Moving transactional context to different thread is not supported for
now".*
Are you sure about *"Moving transactional context to different thread is
not supported for now." *Cause it is working fine in my case and I don't
see any documentation which talks about this. I have used executor service
for parallel processing inside ignite transaction at many places in my
code. Can you please confirm about this?
Thanks,
Prasad
Re: Using Java executor service inside Ignite transaction
Posted by Prasad Bhalerao <pr...@gmail.com>.
Hi,
My business case:
A user can add multiple IPs to subscription to scan the assets. So the
subscription can have N number of IPs.
User can add IPs in following format:
10.10.10.10,
10.10.10.12-10.10.10.100, 10.10.10.102, 10.10.10.105-10.10.11.255
So the Subscription is top level entity. Then there is a Asset group. A
user can create multiple asset group under given subscription. There can be
few thousands asset groups under one subscription.
Asset group is a container for IPs, i.e. a user can add IPs to asset groups
which are present in his subscription and user can launch a scan using
asset groups.
Now when a user removes IPs from subscription, it needs to be removed from
the subscription as well the all asset groups. Now this must happen in a
single transaction. This is the simplest case I have.
What I meant by data processing?
- The add and remove operations are performed on these IPs using tree range
set. After performing add/remove operation I identify the exact Ids or rows
which needs to be added/updated and removed from cache. I call it as
finding delta.
Performing all mentioned operations is data processing for me. I have
created Callable/Runnable tasks to perform these operations.
My affinity key is subscription id. I submit IgniteRunnable task to remove
IPs from subscription. In normal case a subscription can have 100K to 1
Million individual IPs/IP ranges.
An asset group can have 100K to multi million individual IPs and IP
ranges. A subscription can have 1000 to 5000 asset groups.
In side this Ignite task I perform following steps.
a) Start ignite transaction.
b) Remove IPs from subscription in current thread.
c) Then create simple Callable/Runnable tasks to remove IPs (Data
processing) from asset groups in parallel fashion and in this task I am
also doing the cache updates. [This cache updates does not seem to be
working now as it is happening new threads.]
d) Commit the transaction.
*Now my work around to solve this problem:*
*I**n step C instead of updating the cache in callable task, return the
delta (data needs to be updated in cache) to the thread from which ignite
transaction is started and do the cache updates. *
1) Do you think you this approach is ok or do you have any better
suggestion?
As you said, Transaction is always attached to a single thread. If you do
txStart in one thread, and then a cache operation in any other thread, this
operation will not be enlisted.
2) Is there any plan to support multithreaded cache updates inside ignite
transaction?
Thanks,
Prasad
On Fri, Jun 8, 2018 at 5:06 AM vkulichenko <va...@gmail.com>
wrote:
> Prasad,
>
> What is your business use case and what do you mean by processing? Please
> describe in more detail. Are you sure you need to use the transaction here
> in the first place?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Using Java executor service inside Ignite transaction
Posted by vkulichenko <va...@gmail.com>.
Prasad,
What is your business use case and what do you mean by processing? Please
describe in more detail. Are you sure you need to use the transaction here
in the first place?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Using Java executor service inside Ignite transaction
Posted by Prasad Bhalerao <pr...@gmail.com>.
Rephrasing mail to avoid confusion.
So one cannot use executor service or multi threading to update ignite
cache inside ignite transaction.
If this is the case then how do I see/get latest updated values from cache
in other threads which were updated in in a thread from which the
transaction was started?
Is there any plan to support this feature?
If yes, which release?
Is there any other way to speed up the process?
Because I have to process multiple asset groups (more than 1000) inside
single transaction. if I do it in single thread in sequential manner , my
processing will become slow.
I think I have simple way to solve my current problem. Use multithreading
just for data processing. These threads will receive the data( data from
cache ) to be processed from a thread from which transaction was started.
Each thread will process the data and will return the updated data or
result to the thread who initiated transaction as a callable task's return
value. Then the thread which intiated transaction will do the cache
updates.
Thanks,
Prasad
On Thu, Jun 7, 2018, 6:20 AM Prasad Bhalerao <pr...@gmail.com>
wrote:
> Hi,
>
> So one cannot use executor service or multi threading inside ignite
> transaction.
>
> Is there any plan to support this feature?
> If yes, which release?
>
> Is there any other way to speed up the process?
>
> Because I have to process multiple asset groups (more than 1000) inside
> single transaction. if I do it in single thread in sequential manner , my
> processing will become slow.
>
>
> Thank,
> Prasad
>
> On Thu, Jun 7, 2018, 2:08 AM vkulichenko <va...@gmail.com>
> wrote:
>
>> Prasad,
>>
>> Transaction is always attached to a single thread. If you do txStart in
>> one
>> thread, and then a cache operation in any other thread, this operation
>> will
>> not be enlisted.
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: Using Java executor service inside Ignite transaction
Posted by Prasad Bhalerao <pr...@gmail.com>.
Hi,
So one cannot use executor service or multi threading inside ignite
transaction.
Is there any plan to support this feature?
If yes, which release?
Is there any other way to speed up the process?
Because I have to process multiple asset groups (more than 1000) inside
single transaction. if I do it in single thread in sequential manner , my
processing will become slow.
Thank,
Prasad
On Thu, Jun 7, 2018, 2:08 AM vkulichenko <va...@gmail.com>
wrote:
> Prasad,
>
> Transaction is always attached to a single thread. If you do txStart in one
> thread, and then a cache operation in any other thread, this operation will
> not be enlisted.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Using Java executor service inside Ignite transaction
Posted by vkulichenko <va...@gmail.com>.
Prasad,
Transaction is always attached to a single thread. If you do txStart in one
thread, and then a cache operation in any other thread, this operation will
not be enlisted.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Using Java executor service inside Ignite transaction
Posted by Prasad Bhalerao <pr...@gmail.com>.
As per this link
http://apache-ignite-users.70518.x6.nabble.com/Ignite-2-1-0-transaction-restrictions-in-multi-threading-environment-td19007.html
,it is safe to use multithreaded cache updates inside single ignite
transaction.
On Wed, Jun 6, 2018 at 11:00 PM, Prasad Bhalerao <
prasadbhalerao1983@gmail.com> wrote:
> Hi Andrey, I am starting new mail thread as my previous mails were not
> appearing in ignite archive even after sending it multiple times. But I am
> receiving latest group mails.
>
>
> *Question:* Is it safe to use Java executor service inside ignite
> transaction to do parallel processing on same cache?
>
> My affinity key is subscriptionId and one subscription Id can have
> multiple asset groups.
> In my application I submit the ignite runnable task using subscriptionId
> affinity key. Now inside this ignite task I submit Callable tasks per
> assetGroupId to java executor service to process (remove/add and update)
> cache entries in parallel fashion
> This whole thing is happening inside ignite transaction and it works
> perfectly fine.
> Problem occurs only when I update and then read the same entry and update
> it again in same transaction. This is one particular corner case.
>
>
>
> Andrey said "*In your case, task is executed in separate thread
> of executor service which knows no nothing about transaction context and
> starts a new transaction in new context.*
> *Moving transactional context to different thread is not supported for
> now".*
>
> Are you sure about *"Moving transactional context to different thread is
> not supported for now." *Cause it is working fine in my case and I don't
> see any documentation which talks about this. I have used executor service
> for parallel processing inside ignite transaction at many places in my
> code. Can you please confirm about this?
>
>
>
>
> Thanks,
> Prasad
>
Re: Using Java executor service inside Ignite transaction
Posted by vkulichenko <va...@gmail.com>.
Prasad,
Parallel updates in a transaction are not possible. As a matter of fact,
they happen during commit, so doing actual 'put' invocations won't provide
any improvement. Also, you transaction seems to be local, so I don't think
you are going to be faster than you are now. However, if processing can be
moved outside of transaction without breaking the functionality, you
definitely should do that.
In any case, I'm not sure I understand why you need to do this in a single
transaction. My recommendation would be to revisit the architecture and try
to avoid this kind of locking.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Using Java executor service inside Ignite transaction
Posted by Prasad Bhalerao <pr...@gmail.com>.
Hi Val,
Can you please reply?
Thanks,
Prasad
On Wed, Jun 6, 2018, 11:00 PM Prasad Bhalerao <pr...@gmail.com>
wrote:
> Hi Andrey, I am starting new mail thread as my previous mails were not
> appearing in ignite archive even after sending it multiple times. But I am
> receiving latest group mails.
>
>
> *Question:* Is it safe to use Java executor service inside ignite
> transaction to do parallel processing on same cache?
>
> My affinity key is subscriptionId and one subscription Id can have
> multiple asset groups.
> In my application I submit the ignite runnable task using subscriptionId
> affinity key. Now inside this ignite task I submit Callable tasks per
> assetGroupId to java executor service to process (remove/add and update)
> cache entries in parallel fashion
> This whole thing is happening inside ignite transaction and it works
> perfectly fine.
> Problem occurs only when I update and then read the same entry and update
> it again in same transaction. This is one particular corner case.
>
>
>
> Andrey said "*In your case, task is executed in separate thread
> of executor service which knows no nothing about transaction context and
> starts a new transaction in new context.*
> *Moving transactional context to different thread is not supported for
> now".*
>
> Are you sure about *"Moving transactional context to different thread is
> not supported for now." *Cause it is working fine in my case and I don't
> see any documentation which talks about this. I have used executor service
> for parallel processing inside ignite transaction at many places in my
> code. Can you please confirm about this?
>
>
>
>
> Thanks,
> Prasad
>