You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Dmitriy Govorukhin <dm...@gmail.com> on 2018/05/11 14:21:19 UTC
async operation is not fair async
Hi Igniters,
I have a question. Why our async operation in not really async?
GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
operation completed.
This means all async operation in one thread will be executed one by one
but not in parallel. Async operation is not async.
Example for atomic cache
f1=cache.getAsync(key1);
f2=cache.getAsync(key2);
f1 always will be complete before f2.
I want to have the ability run multiple async operations in one thread.
What do you think?
Maybe we can add new cache adapter with fair async operations?
Re: what are the tehcniques to automatically detect changes in Multiple DBs
and automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
Thanks Juan .. I will check on the same
From: "Juan Rodríguez Hortalá" <ju...@gmail.com>
To: user@ignite.apache.org
Date: 07-08-2018 08:44
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
That looks like something you could do with Kafka connect
https://www.confluent.io/product/connectors/ using the jdbc source and the
ignite sink. Just my 2 cents
On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar <de...@tcs.com>
wrote:
Thanks Prasad for your suggestions.
The Legacy sytems are on different platforms and some of them are products
.. so there is no way of implementing any custom logic in such products to
send update messages. The Legacy Systems remain a black-box to us with
only its DB which is accessible for viewing..
Regards
From: "Prasad Bhalerao" <pr...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 16:51
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
Can this back office legacy system send you a DB update message or can you
make this back office system to send you DB update message?
If yes then you can have the Id/primary key, DB operation and table name
in this DB update message.
In your application you use this information to refresh your cache using
read through mechanism.
Thanks,
Prasad
On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <de...@tcs.com>
wrote:
Thanks Denis.
But as I mentioned in earlier mail, the Caches are meant to be Read-Only
(only to be used by Microservices for fetching data).
The Databases are updated by backoffice legacy systems. Hence we cannot
do a Write-through to the DBs via the CacheStore API.
If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
to hear about any challenges/short-comings if any.
Regards
From: Denis Mekhanikov <dm...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 13:18
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then
you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data
into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in
Multiple DBs and automatically push them into Ignite Cache
Posted by Juan Rodríguez Hortalá <ju...@gmail.com>.
That looks like something you could do with Kafka connect
https://www.confluent.io/product/connectors/ using the jdbc source and the
ignite sink. Just my 2 cents
On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar <de...@tcs.com> wrote:
> Thanks Prasad for your suggestions.
>
> The Legacy sytems are on different platforms and some of them are products
> .. so there is no way of implementing any custom logic in such products to
> send update messages. The Legacy Systems remain a black-box to us with only
> its DB which is accessible for viewing..
>
> Regards
>
>
>
>
>
> From: "Prasad Bhalerao" <pr...@gmail.com>
> To: user@ignite.apache.org
> Date: 06-08-2018 16:51
> Subject: Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> ------------------------------
>
>
>
> Can this back office legacy system send you a DB update message or can you
> make this back office system to send you DB update message?
>
> If yes then you can have the Id/primary key, DB operation and table name
> in this DB update message.
>
> In your application you use this information to refresh your cache using
> read through mechanism.
>
> Thanks,
> Prasad
>
> On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <*deepa.kolwalkar@tcs.com*
> <de...@tcs.com>> wrote:
> Thanks Denis.
>
> But as I mentioned in earlier mail, the Caches are meant to be Read-Only
> (only to be used by Microservices for fetching data).
> The Databases are updated by backoffice legacy systems. Hence we cannot
> do a Write-through to the DBs via the CacheStore API.
>
> If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
> to hear about any challenges/short-comings if any.
>
> Regards
>
>
>
> From: Denis Mekhanikov <*dmekhanikov@gmail.com*
> <dm...@gmail.com>>
> To: *user@ignite.apache.org* <us...@ignite.apache.org>
> Date: 06-08-2018 13:18
> Subject: Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> ------------------------------
>
>
>
> There is no such feature in Ignite.
> If you know, how to subscribe for events in the external database, then
> you can implement this logic yourself.
> You just need to perform put into Ignite cache for every insert into the
> external DB.
>
> But the recommended way to do it is to perform writing on Ignite.
> Cache store with *write-through*
> <https://apacheignite.readme.io/v2.6/docs/3rd-party-store#section-read-through-and-write-through> enabled
> will take care of writing the data into the external DB.
>
> Denis
>
> вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <*deepa.kolwalkar@tcs.com*
> <de...@tcs.com>>:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> *http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html*
> <http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html>
> which suggests the following :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> <https://docs.gridgain.com/docs/goldengate-replication>
> ==============
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
So, in a current implementation, how I can perform the real async operation
in one thread? Any workaround?
What can I do if I have event loop thread model?
On Wed, May 16, 2018 at 12:14 PM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> Dmitriy,
>
> I will add technical details to the ticket, however, looks like there is
> still no consensus on how this change should be presented to a user. It
> would be ok if we changed this behavior in Ignite 3.0, but for one of the
> next point releases we have to agree how this should be enabled/disabled
> (or whether we should delay this change to 3.0 at all).
>
> 2018-05-15 22:13 GMT+03:00 Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com>
> :
>
> > Alexey,
> >
> > Any updates?
> >
> > On Mon, May 14, 2018 at 6:19 PM, Dmitriy Govorukhin <
> > dmitriy.govorukhin@gmail.com> wrote:
> >
> > > Alexey,
> > >
> > > Could you please add more description information for this task? [1]
> > > Perhaps, base steps for implementation.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-8475
> > >
> > > On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
> > > alexey.goncharuk@gmail.com> wrote:
> > >
> > >> Another +1 for the true asynchronous approach. I remember a while ago
> > one
> > >> of the Ignite users raised a similar question regarding the *async
> > method
> > >> being blocked on establishing a TCP connection.
> > >>
> > >> As far as deadlocks go, I have a counter-example. Currently, we check
> > the
> > >> thread-local chain only for a single cache, so if I run the following
> > >> code:
> > >> cache1.getAsync(k1);
> > >> cache2.getAsync(k2);
> > >> then the deadlock is still possible, and I did not see a single user
> > >> complaining about unexpected deadlocks. Rather than implementing this
> > >> cross-cache chain (which would probably add another overhead), I would
> > >> make
> > >> it consistent and allow operations to be run in parallel.
> > >>
> > >> There are many use-cases when having true async operations
> dramatically
> > >> improve performance. Consider, for example, a streaming example when
> > keys
> > >> are being pushed by a client to a cluster. Currently, to run effective
> > >> processing, the user will have to use a data streamer with custom keys
> > >> receiver which may be a huge usability downside. Async operations can
> > >> utilize the cluster resources very efficiently.
> > >>
> > >> Finally, if we want to be on the safe side, we can keep the operation
> > >> chain
> > >> inside a transaction. I see absolutely no point in maintaining this
> > chain
> > >> outside of transactions.
> > >>
> > >> --AG
> > >>
> > >> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> > >> dmitriy.govorukhin@gmail.com>
> > >> :
> > >>
> > >> > Andrey,
> > >> >
> > >> > Do you prefer change behavior at runtime?
> > >> > I guess will be better have different methods for getting cache
> > instance
> > >> > with fair and not fair sync.
> > >> >
> > >> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <ag...@apache.org>
> > wrote:
> > >> >
> > >> > > +1 for fair async operations.
> > >> > >
> > >> > > But I don't like idea use withFairSync() method. We added
> xxxAsync()
> > >> > > methods recently and withAsync() is deprecated.
> > >> > >
> > >> > > I think we should just make methods are async in nature and
> provide
> > >> > > ability of switching to the old behaviour using flag or property.
> > >> > >
> > >> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > >> > > <ds...@apache.org> wrote:
> > >> > > > Vladimir,
> > >> > > >
> > >> > > > In general I agree, but I do get greatly *close-minded* (pun
> > >> intended)
> > >> > > > whenever users' code that worked for the past several years all
> > of a
> > >> > > sudden
> > >> > > > gets deadlocked after an upgrade. Making this feature optional
> is
> > >> even
> > >> > > > worse and more confusing. In this case the best action is no
> > action
> > >> at
> > >> > > all.
> > >> > > >
> > >> > > > BTW, would be interesting to find out how Oracle async driver
> > >> behaves
> > >> > in
> > >> > > > this case.
> > >> > > >
> > >> > > > D.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> > >> vozerov@gridgain.com
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Guys,
> > >> > > >>
> > >> > > >> To build a great product we should be open minded and look to
> the
> > >> > > future,
> > >> > > >> not to the past.
> > >> > > >>
> > >> > > >> Dima raised very valid point - why async is not async? Current
> > >> > > programming
> > >> > > >> culture and demanding performance requirements pushes users
> > towards
> > >> > > >> reactive-style programming. I do not want my thread to ever be
> > >> > blocked.
> > >> > > >> Instead, I want to send a number of concurrent commands and
> > >> optionally
> > >> > > >> subscribe to final result. So trully async API makes total
> sense
> > to
> > >> > me.
> > >> > > >>
> > >> > > >> But personally, my primary interest in this area is SQL. Oracle
> > is
> > >> > > >> preparing new async driver. ADBA - async database access. It
> was
> > >> > > presented
> > >> > > >> on recent JavaOne [1]. It is under active development right
> now -
> > >> juse
> > >> > > >> weave through the mailing list [2]. Some prototypes are already
> > >> there
> > >> > > [3].
> > >> > > >> PostgreSQL community even started adopted it [4]!
> > >> > > >>
> > >> > > >> I am not pushing for immediate actions, but at least we should
> > >> > > understand
> > >> > > >> which way the wind is blowing. As a mid-term goals I would
> > propose
> > >> to
> > >> > > >> finally remove thread ID from our PESSIMISTIC transactions to
> > allow
> > >> > for
> > >> > > >> suspend/resume in different threads. And as a next step I would
> > >> think
> > >> > on
> > >> > > >> adopting async cache and SQL APIs.
> > >> > > >>
> > >> > > >> Vladimir.
> > >> > > >>
> > >> > > >> [1]
> > >> > > >> http://www.oracle.com/technetwork/database/
> > >> > > application-development/jdbc/
> > >> > > >> con1491-3961036.pdf
> > >> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > >> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/
> > >> java/AoJ
> > >> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > >> > > >>
> > >> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > >> > > dsetrakyan@apache.org>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > >> > > >> > dmitriy.govorukhin@gmail.com> wrote:
> > >> > > >> >
> > >> > > >> > > I will edit IGNITE-8475, and remove all part that belong to
> > the
> > >> > > public
> > >> > > >> > api.
> > >> > > >> > > Is it acceptable for you?
> > >> > > >> > >
> > >> > > >> >
> > >> > > >> > Everything is acceptable, as long as the public API is safe
> :)
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: async operation is not fair async
Posted by Alexey Goncharuk <al...@gmail.com>.
Dmitriy,
I will add technical details to the ticket, however, looks like there is
still no consensus on how this change should be presented to a user. It
would be ok if we changed this behavior in Ignite 3.0, but for one of the
next point releases we have to agree how this should be enabled/disabled
(or whether we should delay this change to 3.0 at all).
2018-05-15 22:13 GMT+03:00 Dmitriy Govorukhin <dm...@gmail.com>
:
> Alexey,
>
> Any updates?
>
> On Mon, May 14, 2018 at 6:19 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > Alexey,
> >
> > Could you please add more description information for this task? [1]
> > Perhaps, base steps for implementation.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-8475
> >
> > On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com> wrote:
> >
> >> Another +1 for the true asynchronous approach. I remember a while ago
> one
> >> of the Ignite users raised a similar question regarding the *async
> method
> >> being blocked on establishing a TCP connection.
> >>
> >> As far as deadlocks go, I have a counter-example. Currently, we check
> the
> >> thread-local chain only for a single cache, so if I run the following
> >> code:
> >> cache1.getAsync(k1);
> >> cache2.getAsync(k2);
> >> then the deadlock is still possible, and I did not see a single user
> >> complaining about unexpected deadlocks. Rather than implementing this
> >> cross-cache chain (which would probably add another overhead), I would
> >> make
> >> it consistent and allow operations to be run in parallel.
> >>
> >> There are many use-cases when having true async operations dramatically
> >> improve performance. Consider, for example, a streaming example when
> keys
> >> are being pushed by a client to a cluster. Currently, to run effective
> >> processing, the user will have to use a data streamer with custom keys
> >> receiver which may be a huge usability downside. Async operations can
> >> utilize the cluster resources very efficiently.
> >>
> >> Finally, if we want to be on the safe side, we can keep the operation
> >> chain
> >> inside a transaction. I see absolutely no point in maintaining this
> chain
> >> outside of transactions.
> >>
> >> --AG
> >>
> >> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> >> dmitriy.govorukhin@gmail.com>
> >> :
> >>
> >> > Andrey,
> >> >
> >> > Do you prefer change behavior at runtime?
> >> > I guess will be better have different methods for getting cache
> instance
> >> > with fair and not fair sync.
> >> >
> >> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <ag...@apache.org>
> wrote:
> >> >
> >> > > +1 for fair async operations.
> >> > >
> >> > > But I don't like idea use withFairSync() method. We added xxxAsync()
> >> > > methods recently and withAsync() is deprecated.
> >> > >
> >> > > I think we should just make methods are async in nature and provide
> >> > > ability of switching to the old behaviour using flag or property.
> >> > >
> >> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> >> > > <ds...@apache.org> wrote:
> >> > > > Vladimir,
> >> > > >
> >> > > > In general I agree, but I do get greatly *close-minded* (pun
> >> intended)
> >> > > > whenever users' code that worked for the past several years all
> of a
> >> > > sudden
> >> > > > gets deadlocked after an upgrade. Making this feature optional is
> >> even
> >> > > > worse and more confusing. In this case the best action is no
> action
> >> at
> >> > > all.
> >> > > >
> >> > > > BTW, would be interesting to find out how Oracle async driver
> >> behaves
> >> > in
> >> > > > this case.
> >> > > >
> >> > > > D.
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> >> vozerov@gridgain.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> Guys,
> >> > > >>
> >> > > >> To build a great product we should be open minded and look to the
> >> > > future,
> >> > > >> not to the past.
> >> > > >>
> >> > > >> Dima raised very valid point - why async is not async? Current
> >> > > programming
> >> > > >> culture and demanding performance requirements pushes users
> towards
> >> > > >> reactive-style programming. I do not want my thread to ever be
> >> > blocked.
> >> > > >> Instead, I want to send a number of concurrent commands and
> >> optionally
> >> > > >> subscribe to final result. So trully async API makes total sense
> to
> >> > me.
> >> > > >>
> >> > > >> But personally, my primary interest in this area is SQL. Oracle
> is
> >> > > >> preparing new async driver. ADBA - async database access. It was
> >> > > presented
> >> > > >> on recent JavaOne [1]. It is under active development right now -
> >> juse
> >> > > >> weave through the mailing list [2]. Some prototypes are already
> >> there
> >> > > [3].
> >> > > >> PostgreSQL community even started adopted it [4]!
> >> > > >>
> >> > > >> I am not pushing for immediate actions, but at least we should
> >> > > understand
> >> > > >> which way the wind is blowing. As a mid-term goals I would
> propose
> >> to
> >> > > >> finally remove thread ID from our PESSIMISTIC transactions to
> allow
> >> > for
> >> > > >> suspend/resume in different threads. And as a next step I would
> >> think
> >> > on
> >> > > >> adopting async cache and SQL APIs.
> >> > > >>
> >> > > >> Vladimir.
> >> > > >>
> >> > > >> [1]
> >> > > >> http://www.oracle.com/technetwork/database/
> >> > > application-development/jdbc/
> >> > > >> con1491-3961036.pdf
> >> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> >> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/
> >> java/AoJ
> >> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> >> > > >>
> >> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> >> > > dsetrakyan@apache.org>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> >> > > >> > dmitriy.govorukhin@gmail.com> wrote:
> >> > > >> >
> >> > > >> > > I will edit IGNITE-8475, and remove all part that belong to
> the
> >> > > public
> >> > > >> > api.
> >> > > >> > > Is it acceptable for you?
> >> > > >> > >
> >> > > >> >
> >> > > >> > Everything is acceptable, as long as the public API is safe :)
> >> > > >> >
> >> > > >>
> >> > >
> >> >
> >>
> >
> >
>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Alexey,
Any updates?
On Mon, May 14, 2018 at 6:19 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> Alexey,
>
> Could you please add more description information for this task? [1]
> Perhaps, base steps for implementation.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8475
>
> On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
>> Another +1 for the true asynchronous approach. I remember a while ago one
>> of the Ignite users raised a similar question regarding the *async method
>> being blocked on establishing a TCP connection.
>>
>> As far as deadlocks go, I have a counter-example. Currently, we check the
>> thread-local chain only for a single cache, so if I run the following
>> code:
>> cache1.getAsync(k1);
>> cache2.getAsync(k2);
>> then the deadlock is still possible, and I did not see a single user
>> complaining about unexpected deadlocks. Rather than implementing this
>> cross-cache chain (which would probably add another overhead), I would
>> make
>> it consistent and allow operations to be run in parallel.
>>
>> There are many use-cases when having true async operations dramatically
>> improve performance. Consider, for example, a streaming example when keys
>> are being pushed by a client to a cluster. Currently, to run effective
>> processing, the user will have to use a data streamer with custom keys
>> receiver which may be a huge usability downside. Async operations can
>> utilize the cluster resources very efficiently.
>>
>> Finally, if we want to be on the safe side, we can keep the operation
>> chain
>> inside a transaction. I see absolutely no point in maintaining this chain
>> outside of transactions.
>>
>> --AG
>>
>> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
>> dmitriy.govorukhin@gmail.com>
>> :
>>
>> > Andrey,
>> >
>> > Do you prefer change behavior at runtime?
>> > I guess will be better have different methods for getting cache instance
>> > with fair and not fair sync.
>> >
>> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <ag...@apache.org> wrote:
>> >
>> > > +1 for fair async operations.
>> > >
>> > > But I don't like idea use withFairSync() method. We added xxxAsync()
>> > > methods recently and withAsync() is deprecated.
>> > >
>> > > I think we should just make methods are async in nature and provide
>> > > ability of switching to the old behaviour using flag or property.
>> > >
>> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
>> > > <ds...@apache.org> wrote:
>> > > > Vladimir,
>> > > >
>> > > > In general I agree, but I do get greatly *close-minded* (pun
>> intended)
>> > > > whenever users' code that worked for the past several years all of a
>> > > sudden
>> > > > gets deadlocked after an upgrade. Making this feature optional is
>> even
>> > > > worse and more confusing. In this case the best action is no action
>> at
>> > > all.
>> > > >
>> > > > BTW, would be interesting to find out how Oracle async driver
>> behaves
>> > in
>> > > > this case.
>> > > >
>> > > > D.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
>> vozerov@gridgain.com
>> > >
>> > > > wrote:
>> > > >
>> > > >> Guys,
>> > > >>
>> > > >> To build a great product we should be open minded and look to the
>> > > future,
>> > > >> not to the past.
>> > > >>
>> > > >> Dima raised very valid point - why async is not async? Current
>> > > programming
>> > > >> culture and demanding performance requirements pushes users towards
>> > > >> reactive-style programming. I do not want my thread to ever be
>> > blocked.
>> > > >> Instead, I want to send a number of concurrent commands and
>> optionally
>> > > >> subscribe to final result. So trully async API makes total sense to
>> > me.
>> > > >>
>> > > >> But personally, my primary interest in this area is SQL. Oracle is
>> > > >> preparing new async driver. ADBA - async database access. It was
>> > > presented
>> > > >> on recent JavaOne [1]. It is under active development right now -
>> juse
>> > > >> weave through the mailing list [2]. Some prototypes are already
>> there
>> > > [3].
>> > > >> PostgreSQL community even started adopted it [4]!
>> > > >>
>> > > >> I am not pushing for immediate actions, but at least we should
>> > > understand
>> > > >> which way the wind is blowing. As a mid-term goals I would propose
>> to
>> > > >> finally remove thread ID from our PESSIMISTIC transactions to allow
>> > for
>> > > >> suspend/resume in different threads. And as a next step I would
>> think
>> > on
>> > > >> adopting async cache and SQL APIs.
>> > > >>
>> > > >> Vladimir.
>> > > >>
>> > > >> [1]
>> > > >> http://www.oracle.com/technetwork/database/
>> > > application-development/jdbc/
>> > > >> con1491-3961036.pdf
>> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
>> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/
>> java/AoJ
>> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>> > > >>
>> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
>> > > dsetrakyan@apache.org>
>> > > >> wrote:
>> > > >>
>> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
>> > > >> > dmitriy.govorukhin@gmail.com> wrote:
>> > > >> >
>> > > >> > > I will edit IGNITE-8475, and remove all part that belong to the
>> > > public
>> > > >> > api.
>> > > >> > > Is it acceptable for you?
>> > > >> > >
>> > > >> >
>> > > >> > Everything is acceptable, as long as the public API is safe :)
>> > > >> >
>> > > >>
>> > >
>> >
>>
>
>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Alexey,
Could you please add more description information for this task? [1]
Perhaps, base steps for implementation.
[1] https://issues.apache.org/jira/browse/IGNITE-8475
On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:
> Another +1 for the true asynchronous approach. I remember a while ago one
> of the Ignite users raised a similar question regarding the *async method
> being blocked on establishing a TCP connection.
>
> As far as deadlocks go, I have a counter-example. Currently, we check the
> thread-local chain only for a single cache, so if I run the following code:
> cache1.getAsync(k1);
> cache2.getAsync(k2);
> then the deadlock is still possible, and I did not see a single user
> complaining about unexpected deadlocks. Rather than implementing this
> cross-cache chain (which would probably add another overhead), I would make
> it consistent and allow operations to be run in parallel.
>
> There are many use-cases when having true async operations dramatically
> improve performance. Consider, for example, a streaming example when keys
> are being pushed by a client to a cluster. Currently, to run effective
> processing, the user will have to use a data streamer with custom keys
> receiver which may be a huge usability downside. Async operations can
> utilize the cluster resources very efficiently.
>
> Finally, if we want to be on the safe side, we can keep the operation chain
> inside a transaction. I see absolutely no point in maintaining this chain
> outside of transactions.
>
> --AG
>
> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com>
> :
>
> > Andrey,
> >
> > Do you prefer change behavior at runtime?
> > I guess will be better have different methods for getting cache instance
> > with fair and not fair sync.
> >
> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <ag...@apache.org> wrote:
> >
> > > +1 for fair async operations.
> > >
> > > But I don't like idea use withFairSync() method. We added xxxAsync()
> > > methods recently and withAsync() is deprecated.
> > >
> > > I think we should just make methods are async in nature and provide
> > > ability of switching to the old behaviour using flag or property.
> > >
> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > > <ds...@apache.org> wrote:
> > > > Vladimir,
> > > >
> > > > In general I agree, but I do get greatly *close-minded* (pun
> intended)
> > > > whenever users' code that worked for the past several years all of a
> > > sudden
> > > > gets deadlocked after an upgrade. Making this feature optional is
> even
> > > > worse and more confusing. In this case the best action is no action
> at
> > > all.
> > > >
> > > > BTW, would be interesting to find out how Oracle async driver behaves
> > in
> > > > this case.
> > > >
> > > > D.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> vozerov@gridgain.com
> > >
> > > > wrote:
> > > >
> > > >> Guys,
> > > >>
> > > >> To build a great product we should be open minded and look to the
> > > future,
> > > >> not to the past.
> > > >>
> > > >> Dima raised very valid point - why async is not async? Current
> > > programming
> > > >> culture and demanding performance requirements pushes users towards
> > > >> reactive-style programming. I do not want my thread to ever be
> > blocked.
> > > >> Instead, I want to send a number of concurrent commands and
> optionally
> > > >> subscribe to final result. So trully async API makes total sense to
> > me.
> > > >>
> > > >> But personally, my primary interest in this area is SQL. Oracle is
> > > >> preparing new async driver. ADBA - async database access. It was
> > > presented
> > > >> on recent JavaOne [1]. It is under active development right now -
> juse
> > > >> weave through the mailing list [2]. Some prototypes are already
> there
> > > [3].
> > > >> PostgreSQL community even started adopted it [4]!
> > > >>
> > > >> I am not pushing for immediate actions, but at least we should
> > > understand
> > > >> which way the wind is blowing. As a mid-term goals I would propose
> to
> > > >> finally remove thread ID from our PESSIMISTIC transactions to allow
> > for
> > > >> suspend/resume in different threads. And as a next step I would
> think
> > on
> > > >> adopting async cache and SQL APIs.
> > > >>
> > > >> Vladimir.
> > > >>
> > > >> [1]
> > > >> http://www.oracle.com/technetwork/database/
> > > application-development/jdbc/
> > > >> con1491-3961036.pdf
> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/
> master/java/AoJ
> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > > >>
> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > >> wrote:
> > > >>
> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > > >> > dmitriy.govorukhin@gmail.com> wrote:
> > > >> >
> > > >> > > I will edit IGNITE-8475, and remove all part that belong to the
> > > public
> > > >> > api.
> > > >> > > Is it acceptable for you?
> > > >> > >
> > > >> >
> > > >> > Everything is acceptable, as long as the public API is safe :)
> > > >> >
> > > >>
> > >
> >
>
Re: what are the tehcniques to automatically detect changes in Multiple DBs
and automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
Thanks Prasad for your suggestions.
The Legacy sytems are on different platforms and some of them are products
.. so there is no way of implementing any custom logic in such products to
send update messages. The Legacy Systems remain a black-box to us with
only its DB which is accessible for viewing..
Regards
From: "Prasad Bhalerao" <pr...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 16:51
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
Can this back office legacy system send you a DB update message or can you
make this back office system to send you DB update message?
If yes then you can have the Id/primary key, DB operation and table name
in this DB update message.
In your application you use this information to refresh your cache using
read through mechanism.
Thanks,
Prasad
On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <de...@tcs.com>
wrote:
Thanks Denis.
But as I mentioned in earlier mail, the Caches are meant to be Read-Only
(only to be used by Microservices for fetching data).
The Databases are updated by backoffice legacy systems. Hence we cannot
do a Write-through to the DBs via the CacheStore API.
If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
to hear about any challenges/short-comings if any.
Regards
From: Denis Mekhanikov <dm...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 13:18
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then
you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data
into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in
Multiple DBs and automatically push them into Ignite Cache
Posted by Prasad Bhalerao <pr...@gmail.com>.
Can this back office legacy system send you a DB update message or can you
make this back office system to send you DB update message?
If yes then you can have the Id/primary key, DB operation and table name in
this DB update message.
In your application you use this information to refresh your cache using
read through mechanism.
Thanks,
Prasad
On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <de...@tcs.com>
wrote:
> Thanks Denis.
>
> But as I mentioned in earlier mail, the Caches are meant to be Read-Only
> (only to be used by Microservices for fetching data).
> The Databases are updated by backoffice legacy systems. Hence we cannot
> do a Write-through to the DBs via the CacheStore API.
>
> If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
> to hear about any challenges/short-comings if any.
>
> Regards
>
>
>
> From: Denis Mekhanikov <dm...@gmail.com>
> To: user@ignite.apache.org
> Date: 06-08-2018 13:18
> Subject: Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> ------------------------------
>
>
>
> There is no such feature in Ignite.
> If you know, how to subscribe for events in the external database, then
> you can implement this logic yourself.
> You just need to perform put into Ignite cache for every insert into the
> external DB.
>
> But the recommended way to do it is to perform writing on Ignite.
> Cache store with *write-through*
> <https://apacheignite.readme.io/v2.6/docs/3rd-party-store#section-read-through-and-write-through> enabled
> will take care of writing the data into the external DB.
>
> Denis
>
> вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <*deepa.kolwalkar@tcs.com*
> <de...@tcs.com>>:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> *http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html*
> <http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html>
> which suggests the following :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> <https://docs.gridgain.com/docs/goldengate-replication>
> ==============
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
Re: what are the tehcniques to automatically detect changes in Multiple DBs
and automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
Thanks Denis.
But as I mentioned in earlier mail, the Caches are meant to be Read-Only
(only to be used by Microservices for fetching data).
The Databases are updated by backoffice legacy systems. Hence we cannot
do a Write-through to the DBs via the CacheStore API.
If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
to hear about any challenges/short-comings if any.
Regards
From: Denis Mekhanikov <dm...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 13:18
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then
you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data
into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in
Multiple DBs and automatically push them into Ignite Cache
Posted by Denis Mekhanikov <dm...@gmail.com>.
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you
can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through
<https://apacheignite.readme.io/v2.6/docs/3rd-party-store#section-read-through-and-write-through>
enabled
will take care of writing the data into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
> which suggests the following :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> <https://docs.gridgain.com/docs/goldengate-replication>
> ==============
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
>
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
what are the tehcniques to automatically detect changes in Multiple DBs and
automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: can Ignite Cache automatically detect changes in multiple
Underlying DBs and stay in sync with them
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi,
if this question is not related to some contribution than, probably, better
place to ask is user@ignite.apache.org mailing list.
Sincerely,
Dmitriy Pavlov
пт, 3 авг. 2018 г. в 11:44, Deepa Kolwalkar <de...@tcs.com>:
> Hi
>
> While searching for a solution on how to have DB updates to reflect
> automatically in Ignite Cache, I came across this post dt Mar-2018 which
> states that :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> https://docs.gridgain.com/docs/goldengate-replication
> ==============
>
> http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>
Re: can Ignite Cache automatically detect changes in multiple
Underlying DBs and stay in sync with them
Posted by Denis Magda <dm...@apache.org>.
Hi Deepa,
What is your use case? There might be a solution.
--
Denis
On Fri, Aug 3, 2018 at 1:44 AM Deepa Kolwalkar <de...@tcs.com>
wrote:
> Hi
>
> While searching for a solution on how to have DB updates to reflect
> automatically in Ignite Cache, I came across this post dt Mar-2018 which
> states that :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> https://docs.gridgain.com/docs/goldengate-replication
> ==============
>
> http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>
can Ignite Cache automatically detect changes in multiple Underlying DBs
and stay in sync with them
Posted by Deepa Kolwalkar <de...@tcs.com>.
Hi
While searching for a solution on how to have DB updates to reflect
automatically in Ignite Cache, I came across this post dt Mar-2018 which
states that :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Folks,
Any comments?
I will start to implement withFairAsync(); decorator soon.
On Wed, Aug 1, 2018 at 12:22 PM Dmitriy Pavlov <dp...@gmail.com>
wrote:
> Igniters,
>
> I've re-read this thread in brief. As far as I know this change is not
> coming from some company, so this change will be at least good for healthy
> community building.
>
> And I didn't find any obstacles why not to implement approach with new mode
> .withFairAsync() for cases user is completely aware of consequences.
>
> It could be not public API to avoid anyone will use it. It could be
> used,e.g. in integrations and by qualified users to gain as much
> throutghput as it is possible.
>
> So I would like to be an sponsor here. If anyone or Dmitriy G. will
> contribute this change, I will merge it. I hope PMCs are agree here and
> will not veto this change.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 24 мая 2018 г. в 15:13, Yakov Zhdanov <yz...@apache.org>:
>
> > Alexey Goncharuk, I remember we started working on async connection
> > establishment. This should fix latency issue related to network which I
> > believe gives the most contribution to overall latency. Mapping logic and
> > other stuff can be ignored as it can very rarely be an issue at least on
> > stable tolopogies. What is the status with async connections? That would
> > really be a huge improvement!
> >
> > Also please remember that uncontrolled async operations may lead to OOME,
> > therefore at some point when there are too many uncompleted async
> > operations newly invoked async operations should become synchronous, i.e.
> > we should return completed future, ignoring the fact that user expected
> us
> > to be async.
> >
> > I would like to have very strong reasons to start reapproaching this.
> >
> > --Yakov
> >
>
Re: async operation is not fair async
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Igniters,
I've re-read this thread in brief. As far as I know this change is not
coming from some company, so this change will be at least good for healthy
community building.
And I didn't find any obstacles why not to implement approach with new mode
.withFairAsync() for cases user is completely aware of consequences.
It could be not public API to avoid anyone will use it. It could be
used,e.g. in integrations and by qualified users to gain as much
throutghput as it is possible.
So I would like to be an sponsor here. If anyone or Dmitriy G. will
contribute this change, I will merge it. I hope PMCs are agree here and
will not veto this change.
Sincerely,
Dmitriy Pavlov
чт, 24 мая 2018 г. в 15:13, Yakov Zhdanov <yz...@apache.org>:
> Alexey Goncharuk, I remember we started working on async connection
> establishment. This should fix latency issue related to network which I
> believe gives the most contribution to overall latency. Mapping logic and
> other stuff can be ignored as it can very rarely be an issue at least on
> stable tolopogies. What is the status with async connections? That would
> really be a huge improvement!
>
> Also please remember that uncontrolled async operations may lead to OOME,
> therefore at some point when there are too many uncompleted async
> operations newly invoked async operations should become synchronous, i.e.
> we should return completed future, ignoring the fact that user expected us
> to be async.
>
> I would like to have very strong reasons to start reapproaching this.
>
> --Yakov
>
Re: async operation is not fair async
Posted by Yakov Zhdanov <yz...@apache.org>.
Alexey Goncharuk, I remember we started working on async connection
establishment. This should fix latency issue related to network which I
believe gives the most contribution to overall latency. Mapping logic and
other stuff can be ignored as it can very rarely be an issue at least on
stable tolopogies. What is the status with async connections? That would
really be a huge improvement!
Also please remember that uncontrolled async operations may lead to OOME,
therefore at some point when there are too many uncompleted async
operations newly invoked async operations should become synchronous, i.e.
we should return completed future, ignoring the fact that user expected us
to be async.
I would like to have very strong reasons to start reapproaching this.
--Yakov
Re: async operation is not fair async
Posted by Alexey Goncharuk <al...@gmail.com>.
Another +1 for the true asynchronous approach. I remember a while ago one
of the Ignite users raised a similar question regarding the *async method
being blocked on establishing a TCP connection.
As far as deadlocks go, I have a counter-example. Currently, we check the
thread-local chain only for a single cache, so if I run the following code:
cache1.getAsync(k1);
cache2.getAsync(k2);
then the deadlock is still possible, and I did not see a single user
complaining about unexpected deadlocks. Rather than implementing this
cross-cache chain (which would probably add another overhead), I would make
it consistent and allow operations to be run in parallel.
There are many use-cases when having true async operations dramatically
improve performance. Consider, for example, a streaming example when keys
are being pushed by a client to a cluster. Currently, to run effective
processing, the user will have to use a data streamer with custom keys
receiver which may be a huge usability downside. Async operations can
utilize the cluster resources very efficiently.
Finally, if we want to be on the safe side, we can keep the operation chain
inside a transaction. I see absolutely no point in maintaining this chain
outside of transactions.
--AG
2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <dm...@gmail.com>
:
> Andrey,
>
> Do you prefer change behavior at runtime?
> I guess will be better have different methods for getting cache instance
> with fair and not fair sync.
>
> On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <ag...@apache.org> wrote:
>
> > +1 for fair async operations.
> >
> > But I don't like idea use withFairSync() method. We added xxxAsync()
> > methods recently and withAsync() is deprecated.
> >
> > I think we should just make methods are async in nature and provide
> > ability of switching to the old behaviour using flag or property.
> >
> > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > <ds...@apache.org> wrote:
> > > Vladimir,
> > >
> > > In general I agree, but I do get greatly *close-minded* (pun intended)
> > > whenever users' code that worked for the past several years all of a
> > sudden
> > > gets deadlocked after an upgrade. Making this feature optional is even
> > > worse and more confusing. In this case the best action is no action at
> > all.
> > >
> > > BTW, would be interesting to find out how Oracle async driver behaves
> in
> > > this case.
> > >
> > > D.
> > >
> > >
> > >
> > >
> > >
> > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > >> Guys,
> > >>
> > >> To build a great product we should be open minded and look to the
> > future,
> > >> not to the past.
> > >>
> > >> Dima raised very valid point - why async is not async? Current
> > programming
> > >> culture and demanding performance requirements pushes users towards
> > >> reactive-style programming. I do not want my thread to ever be
> blocked.
> > >> Instead, I want to send a number of concurrent commands and optionally
> > >> subscribe to final result. So trully async API makes total sense to
> me.
> > >>
> > >> But personally, my primary interest in this area is SQL. Oracle is
> > >> preparing new async driver. ADBA - async database access. It was
> > presented
> > >> on recent JavaOne [1]. It is under active development right now - juse
> > >> weave through the mailing list [2]. Some prototypes are already there
> > [3].
> > >> PostgreSQL community even started adopted it [4]!
> > >>
> > >> I am not pushing for immediate actions, but at least we should
> > understand
> > >> which way the wind is blowing. As a mid-term goals I would propose to
> > >> finally remove thread ID from our PESSIMISTIC transactions to allow
> for
> > >> suspend/resume in different threads. And as a next step I would think
> on
> > >> adopting async cache and SQL APIs.
> > >>
> > >> Vladimir.
> > >>
> > >> [1]
> > >> http://www.oracle.com/technetwork/database/
> > application-development/jdbc/
> > >> con1491-3961036.pdf
> > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > >>
> > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > >> wrote:
> > >>
> > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > >> > dmitriy.govorukhin@gmail.com> wrote:
> > >> >
> > >> > > I will edit IGNITE-8475, and remove all part that belong to the
> > public
> > >> > api.
> > >> > > Is it acceptable for you?
> > >> > >
> > >> >
> > >> > Everything is acceptable, as long as the public API is safe :)
> > >> >
> > >>
> >
>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Andrey,
Do you prefer change behavior at runtime?
I guess will be better have different methods for getting cache instance
with fair and not fair sync.
On Mon, May 14, 2018 at 3:39 PM, Andrey Gura <ag...@apache.org> wrote:
> +1 for fair async operations.
>
> But I don't like idea use withFairSync() method. We added xxxAsync()
> methods recently and withAsync() is deprecated.
>
> I think we should just make methods are async in nature and provide
> ability of switching to the old behaviour using flag or property.
>
> On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> <ds...@apache.org> wrote:
> > Vladimir,
> >
> > In general I agree, but I do get greatly *close-minded* (pun intended)
> > whenever users' code that worked for the past several years all of a
> sudden
> > gets deadlocked after an upgrade. Making this feature optional is even
> > worse and more confusing. In this case the best action is no action at
> all.
> >
> > BTW, would be interesting to find out how Oracle async driver behaves in
> > this case.
> >
> > D.
> >
> >
> >
> >
> >
> > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> >> Guys,
> >>
> >> To build a great product we should be open minded and look to the
> future,
> >> not to the past.
> >>
> >> Dima raised very valid point - why async is not async? Current
> programming
> >> culture and demanding performance requirements pushes users towards
> >> reactive-style programming. I do not want my thread to ever be blocked.
> >> Instead, I want to send a number of concurrent commands and optionally
> >> subscribe to final result. So trully async API makes total sense to me.
> >>
> >> But personally, my primary interest in this area is SQL. Oracle is
> >> preparing new async driver. ADBA - async database access. It was
> presented
> >> on recent JavaOne [1]. It is under active development right now - juse
> >> weave through the mailing list [2]. Some prototypes are already there
> [3].
> >> PostgreSQL community even started adopted it [4]!
> >>
> >> I am not pushing for immediate actions, but at least we should
> understand
> >> which way the wind is blowing. As a mid-term goals I would propose to
> >> finally remove thread ID from our PESSIMISTIC transactions to allow for
> >> suspend/resume in different threads. And as a next step I would think on
> >> adopting async cache and SQL APIs.
> >>
> >> Vladimir.
> >>
> >> [1]
> >> http://www.oracle.com/technetwork/database/
> application-development/jdbc/
> >> con1491-3961036.pdf
> >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> >> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> >>
> >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> >> wrote:
> >>
> >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> >> > dmitriy.govorukhin@gmail.com> wrote:
> >> >
> >> > > I will edit IGNITE-8475, and remove all part that belong to the
> public
> >> > api.
> >> > > Is it acceptable for you?
> >> > >
> >> >
> >> > Everything is acceptable, as long as the public API is safe :)
> >> >
> >>
>
Re: async operation is not fair async
Posted by Andrey Gura <ag...@apache.org>.
+1 for fair async operations.
But I don't like idea use withFairSync() method. We added xxxAsync()
methods recently and withAsync() is deprecated.
I think we should just make methods are async in nature and provide
ability of switching to the old behaviour using flag or property.
On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
<ds...@apache.org> wrote:
> Vladimir,
>
> In general I agree, but I do get greatly *close-minded* (pun intended)
> whenever users' code that worked for the past several years all of a sudden
> gets deadlocked after an upgrade. Making this feature optional is even
> worse and more confusing. In this case the best action is no action at all.
>
> BTW, would be interesting to find out how Oracle async driver behaves in
> this case.
>
> D.
>
>
>
>
>
> On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
>> Guys,
>>
>> To build a great product we should be open minded and look to the future,
>> not to the past.
>>
>> Dima raised very valid point - why async is not async? Current programming
>> culture and demanding performance requirements pushes users towards
>> reactive-style programming. I do not want my thread to ever be blocked.
>> Instead, I want to send a number of concurrent commands and optionally
>> subscribe to final result. So trully async API makes total sense to me.
>>
>> But personally, my primary interest in this area is SQL. Oracle is
>> preparing new async driver. ADBA - async database access. It was presented
>> on recent JavaOne [1]. It is under active development right now - juse
>> weave through the mailing list [2]. Some prototypes are already there [3].
>> PostgreSQL community even started adopted it [4]!
>>
>> I am not pushing for immediate actions, but at least we should understand
>> which way the wind is blowing. As a mid-term goals I would propose to
>> finally remove thread ID from our PESSIMISTIC transactions to allow for
>> suspend/resume in different threads. And as a next step I would think on
>> adopting async cache and SQL APIs.
>>
>> Vladimir.
>>
>> [1]
>> http://www.oracle.com/technetwork/database/application-development/jdbc/
>> con1491-3961036.pdf
>> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
>> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
>> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>>
>> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>>
>> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
>> > dmitriy.govorukhin@gmail.com> wrote:
>> >
>> > > I will edit IGNITE-8475, and remove all part that belong to the public
>> > api.
>> > > Is it acceptable for you?
>> > >
>> >
>> > Everything is acceptable, as long as the public API is safe :)
>> >
>>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Vladimir,
In general I agree, but I do get greatly *close-minded* (pun intended)
whenever users' code that worked for the past several years all of a sudden
gets deadlocked after an upgrade. Making this feature optional is even
worse and more confusing. In this case the best action is no action at all.
BTW, would be interesting to find out how Oracle async driver behaves in
this case.
D.
On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Guys,
>
> To build a great product we should be open minded and look to the future,
> not to the past.
>
> Dima raised very valid point - why async is not async? Current programming
> culture and demanding performance requirements pushes users towards
> reactive-style programming. I do not want my thread to ever be blocked.
> Instead, I want to send a number of concurrent commands and optionally
> subscribe to final result. So trully async API makes total sense to me.
>
> But personally, my primary interest in this area is SQL. Oracle is
> preparing new async driver. ADBA - async database access. It was presented
> on recent JavaOne [1]. It is under active development right now - juse
> weave through the mailing list [2]. Some prototypes are already there [3].
> PostgreSQL community even started adopted it [4]!
>
> I am not pushing for immediate actions, but at least we should understand
> which way the wind is blowing. As a mid-term goals I would propose to
> finally remove thread ID from our PESSIMISTIC transactions to allow for
> suspend/resume in different threads. And as a next step I would think on
> adopting async cache and SQL APIs.
>
> Vladimir.
>
> [1]
> http://www.oracle.com/technetwork/database/application-development/jdbc/
> con1491-3961036.pdf
> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>
> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > dmitriy.govorukhin@gmail.com> wrote:
> >
> > > I will edit IGNITE-8475, and remove all part that belong to the public
> > api.
> > > Is it acceptable for you?
> > >
> >
> > Everything is acceptable, as long as the public API is safe :)
> >
>
Re: async operation is not fair async
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Guys,
To build a great product we should be open minded and look to the future,
not to the past.
Dima raised very valid point - why async is not async? Current programming
culture and demanding performance requirements pushes users towards
reactive-style programming. I do not want my thread to ever be blocked.
Instead, I want to send a number of concurrent commands and optionally
subscribe to final result. So trully async API makes total sense to me.
But personally, my primary interest in this area is SQL. Oracle is
preparing new async driver. ADBA - async database access. It was presented
on recent JavaOne [1]. It is under active development right now - juse
weave through the mailing list [2]. Some prototypes are already there [3].
PostgreSQL community even started adopted it [4]!
I am not pushing for immediate actions, but at least we should understand
which way the wind is blowing. As a mid-term goals I would propose to
finally remove thread ID from our PESSIMISTIC transactions to allow for
suspend/resume in different threads. And as a next step I would think on
adopting async cache and SQL APIs.
Vladimir.
[1]
http://www.oracle.com/technetwork/database/application-development/jdbc/con1491-3961036.pdf
[2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
[3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
[4] https://github.com/pgjdbc/pgjdbc/issues/978
On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > I will edit IGNITE-8475, and remove all part that belong to the public
> api.
> > Is it acceptable for you?
> >
>
> Everything is acceptable, as long as the public API is safe :)
>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> I will edit IGNITE-8475, and remove all part that belong to the public api.
> Is it acceptable for you?
>
Everything is acceptable, as long as the public API is safe :)
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
I will edit IGNITE-8475, and remove all part that belong to the public api.
Is it acceptable for you?
пт, 11 мая 2018 г., 21:29 Dmitriy Setrakyan <ds...@apache.org>:
> On Fri, May 11, 2018 at 7:23 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > Dmitriy S,
> >
> > If it will be in the internal package, and only for internal usage, are
> you
> > agree with changes?
> >
>
> Yes, but please be careful not to create deadlocks for ourselves.
>
> Can you please close the ticket?
>
> D.
>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, May 11, 2018 at 7:23 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> Dmitriy S,
>
> If it will be in the internal package, and only for internal usage, are you
> agree with changes?
>
Yes, but please be careful not to create deadlocks for ourselves.
Can you please close the ticket?
D.
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Dmitriy S,
If it will be in the internal package, and only for internal usage, are you
agree with changes?
пт, 11 мая 2018 г., 21:12 Dmitriy Setrakyan <ds...@apache.org>:
> On Fri, May 11, 2018 at 6:49 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > Dmitriy S,
> >
> > It is not broke existing code, because for use this ability you must use
> > decorator "withFairSycn()".
> >
> > What about the argument of Vladimir?
> >
>
> Here is Vladimir's quote:
>
>
> *This would also be helpful for transactional SQL as it would allow to
> hide**network
> latency. But there is a problem - deadlocks. We need to inform user*
> *that this mode should be used with great care. *
>
>
> I would rather not change anything instead of increasing the probability of
> deadlocks. This was the main reason for the current behavior to begin with.
>
> In my view, if something is needed for the transactional SQL, please add it
> internally. Let's not corrupt the public API by adding dangerous methods.
>
> D.
>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, May 11, 2018 at 6:49 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> Dmitriy S,
>
> It is not broke existing code, because for use this ability you must use
> decorator "withFairSycn()".
>
> What about the argument of Vladimir?
>
Here is Vladimir's quote:
*This would also be helpful for transactional SQL as it would allow to
hide**network
latency. But there is a problem - deadlocks. We need to inform user*
*that this mode should be used with great care. *
I would rather not change anything instead of increasing the probability of
deadlocks. This was the main reason for the current behavior to begin with.
In my view, if something is needed for the transactional SQL, please add it
internally. Let's not corrupt the public API by adding dangerous methods.
D.
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Dmitriy S,
It is not broke existing code, because for use this ability you must use
decorator "withFairSycn()".
What about the argument of Vladimir?
On Fri, May 11, 2018 at 8:41 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Fri, May 11, 2018 at 6:38 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > Dmitriy S,
> >
> > Why method named as "async" but does not work as async? This is
> misleading.
> >
> > getAllAsync() is a special case. Not always you can use getAllAsync()
> > instead
> > of multiple getAsync().
> > In this topic, I wanna discuss problem not only for the GET operation but
> > also all async operation behavior in the one thread.
> >
> > In compute grid we can run multiple async compute operation in the one
> > thread, so why we can not do this for cache?
>
>
> Because it will break a lot of existing code and create bugs we cannot even
> predict at this point. I am not sure why has this become a problem. Is it
> preventing us from accomplishing some other task? If not, then I propose to
> drop it.
>
> D.
>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, May 11, 2018 at 6:38 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> Dmitriy S,
>
> Why method named as "async" but does not work as async? This is misleading.
>
> getAllAsync() is a special case. Not always you can use getAllAsync()
> instead
> of multiple getAsync().
> In this topic, I wanna discuss problem not only for the GET operation but
> also all async operation behavior in the one thread.
>
> In compute grid we can run multiple async compute operation in the one
> thread, so why we can not do this for cache?
Because it will break a lot of existing code and create bugs we cannot even
predict at this point. I am not sure why has this become a problem. Is it
preventing us from accomplishing some other task? If not, then I propose to
drop it.
D.
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Dmitriy S,
Why method named as "async" but does not work as async? This is misleading.
getAllAsync() is a special case. Not always you can use getAllAsync() instead
of multiple getAsync().
In this topic, I wanna discuss problem not only for the GET operation but
also all async operation behavior in the one thread.
In compute grid we can run multiple async compute operation in the one
thread, so why we can not do this for cache?
On Fri, May 11, 2018 at 8:33 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Fri, May 11, 2018 at 6:29 PM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > IMO you can complete async operations one before another if these
> > operations are related to independent data.
> >
> > It is strange why Ignite users are not confused by current API. So I
> > support Dmitriy's G. suggestion.
> >
>
> Again, this is a solution looking for a problem. Nobody complains about
> this, so there really isn't any issue. There are so many other tasks we
> could focus on. Let's not fix what's not broken.
>
> D.
>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, May 11, 2018 at 6:29 PM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> IMO you can complete async operations one before another if these
> operations are related to independent data.
>
> It is strange why Ignite users are not confused by current API. So I
> support Dmitriy's G. suggestion.
>
Again, this is a solution looking for a problem. Nobody complains about
this, so there really isn't any issue. There are so many other tasks we
could focus on. Let's not fix what's not broken.
D.
Re: async operation is not fair async
Posted by Dmitry Pavlov <dp...@gmail.com>.
IMO you can complete async operations one before another if these
operations are related to independent data.
It is strange why Ignite users are not confused by current API. So I
support Dmitriy's G. suggestion.
пт, 11 мая 2018 г. в 20:24, Dmitriy Setrakyan <ds...@apache.org>:
> Guys,
>
> I am not sure I like this approach, especially for this code:
>
> f1=cache.getAsync(key1);
> f2=cache.getAsync(key2);
>
> You cannot complete f2 before f1. If you do, the code is unusable and it is
> impossible to predict anything. If you need to get 2 elements
> asynchronously, use getAllAsync() instead.
>
> Let's not introduce such usability issues into our API and stop trying to
> fix what's not broken. I propose to close the ticket as "Won't Fix".
>
> D.
>
> On Fri, May 11, 2018 at 6:14 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > Igniters,
> >
> > I created the issue. IGNITE-8475
> > <https://issues.apache.org/jira/browse/IGNITE-8475>
> >
> > Any comments are welcome.
> >
> > On Fri, May 11, 2018 at 6:32 PM, Ivan Rakov <iv...@gmail.com>
> wrote:
> >
> > > Agree. "fair" is more descriptive.
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > >
> > > On 11.05.2018 18:30, Dmitriy Govorukhin wrote:
> > >
> > >> Ivan,
> > >>
> > >> My suggestion "withFairAsync()". What do you think?
> > >>
> > >> On Fri, May 11, 2018 at 6:23 PM, Ivan Rakov <iv...@gmail.com>
> > >> wrote:
> > >>
> > >> I think, the best option from API side is to add decorating
> > >>> withExplicitAsync() method.
> > >>> We already have withKeepBinary, withExpiryPolicy and so on.
> > >>>
> > >>> Best Regards,
> > >>> Ivan Rakov
> > >>>
> > >>>
> > >>> On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
> > >>>
> > >>> Vladimir,
> > >>>>
> > >>>> Should we create the new cache adapter or rework GridCacheAdapter?
> > >>>>
> > >>>> On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <
> > vozerov@gridgain.com>
> > >>>> wrote:
> > >>>>
> > >>>> +1
> > >>>>
> > >>>>> This would also be helpful for transactional SQL as it would allow
> to
> > >>>>> hide
> > >>>>> network latency. But there is a problem - deadlocks. We need to
> > inform
> > >>>>> user
> > >>>>> that this mode should be used with great care.
> > >>>>>
> > >>>>> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
> > >>>>> dmitriy.govorukhin@gmail.com> wrote:
> > >>>>>
> > >>>>> Hi Igniters,
> > >>>>>
> > >>>>>> I have a question. Why our async operation in not really async?
> > >>>>>>
> > >>>>>> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last
> > async
> > >>>>>> operation completed.
> > >>>>>>
> > >>>>>> This means all async operation in one thread will be executed one
> by
> > >>>>>> one
> > >>>>>> but not in parallel. Async operation is not async.
> > >>>>>>
> > >>>>>> Example for atomic cache
> > >>>>>>
> > >>>>>> f1=cache.getAsync(key1);
> > >>>>>> f2=cache.getAsync(key2);
> > >>>>>>
> > >>>>>> f1 always will be complete before f2.
> > >>>>>>
> > >>>>>> I want to have the ability run multiple async operations in one
> > >>>>>> thread.
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> Maybe we can add new cache adapter with fair async operations?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >
> >
>
Re: async operation is not fair async
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Guys,
I am not sure I like this approach, especially for this code:
f1=cache.getAsync(key1);
f2=cache.getAsync(key2);
You cannot complete f2 before f1. If you do, the code is unusable and it is
impossible to predict anything. If you need to get 2 elements
asynchronously, use getAllAsync() instead.
Let's not introduce such usability issues into our API and stop trying to
fix what's not broken. I propose to close the ticket as "Won't Fix".
D.
On Fri, May 11, 2018 at 6:14 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> Igniters,
>
> I created the issue. IGNITE-8475
> <https://issues.apache.org/jira/browse/IGNITE-8475>
>
> Any comments are welcome.
>
> On Fri, May 11, 2018 at 6:32 PM, Ivan Rakov <iv...@gmail.com> wrote:
>
> > Agree. "fair" is more descriptive.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 11.05.2018 18:30, Dmitriy Govorukhin wrote:
> >
> >> Ivan,
> >>
> >> My suggestion "withFairAsync()". What do you think?
> >>
> >> On Fri, May 11, 2018 at 6:23 PM, Ivan Rakov <iv...@gmail.com>
> >> wrote:
> >>
> >> I think, the best option from API side is to add decorating
> >>> withExplicitAsync() method.
> >>> We already have withKeepBinary, withExpiryPolicy and so on.
> >>>
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>>
> >>> On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
> >>>
> >>> Vladimir,
> >>>>
> >>>> Should we create the new cache adapter or rework GridCacheAdapter?
> >>>>
> >>>> On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <
> vozerov@gridgain.com>
> >>>> wrote:
> >>>>
> >>>> +1
> >>>>
> >>>>> This would also be helpful for transactional SQL as it would allow to
> >>>>> hide
> >>>>> network latency. But there is a problem - deadlocks. We need to
> inform
> >>>>> user
> >>>>> that this mode should be used with great care.
> >>>>>
> >>>>> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
> >>>>> dmitriy.govorukhin@gmail.com> wrote:
> >>>>>
> >>>>> Hi Igniters,
> >>>>>
> >>>>>> I have a question. Why our async operation in not really async?
> >>>>>>
> >>>>>> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last
> async
> >>>>>> operation completed.
> >>>>>>
> >>>>>> This means all async operation in one thread will be executed one by
> >>>>>> one
> >>>>>> but not in parallel. Async operation is not async.
> >>>>>>
> >>>>>> Example for atomic cache
> >>>>>>
> >>>>>> f1=cache.getAsync(key1);
> >>>>>> f2=cache.getAsync(key2);
> >>>>>>
> >>>>>> f1 always will be complete before f2.
> >>>>>>
> >>>>>> I want to have the ability run multiple async operations in one
> >>>>>> thread.
> >>>>>> What do you think?
> >>>>>>
> >>>>>> Maybe we can add new cache adapter with fair async operations?
> >>>>>>
> >>>>>>
> >>>>>>
> >
>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Igniters,
I created the issue. IGNITE-8475
<https://issues.apache.org/jira/browse/IGNITE-8475>
Any comments are welcome.
On Fri, May 11, 2018 at 6:32 PM, Ivan Rakov <iv...@gmail.com> wrote:
> Agree. "fair" is more descriptive.
>
> Best Regards,
> Ivan Rakov
>
>
> On 11.05.2018 18:30, Dmitriy Govorukhin wrote:
>
>> Ivan,
>>
>> My suggestion "withFairAsync()". What do you think?
>>
>> On Fri, May 11, 2018 at 6:23 PM, Ivan Rakov <iv...@gmail.com>
>> wrote:
>>
>> I think, the best option from API side is to add decorating
>>> withExplicitAsync() method.
>>> We already have withKeepBinary, withExpiryPolicy and so on.
>>>
>>> Best Regards,
>>> Ivan Rakov
>>>
>>>
>>> On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
>>>
>>> Vladimir,
>>>>
>>>> Should we create the new cache adapter or rework GridCacheAdapter?
>>>>
>>>> On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <vo...@gridgain.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>>> This would also be helpful for transactional SQL as it would allow to
>>>>> hide
>>>>> network latency. But there is a problem - deadlocks. We need to inform
>>>>> user
>>>>> that this mode should be used with great care.
>>>>>
>>>>> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
>>>>> dmitriy.govorukhin@gmail.com> wrote:
>>>>>
>>>>> Hi Igniters,
>>>>>
>>>>>> I have a question. Why our async operation in not really async?
>>>>>>
>>>>>> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
>>>>>> operation completed.
>>>>>>
>>>>>> This means all async operation in one thread will be executed one by
>>>>>> one
>>>>>> but not in parallel. Async operation is not async.
>>>>>>
>>>>>> Example for atomic cache
>>>>>>
>>>>>> f1=cache.getAsync(key1);
>>>>>> f2=cache.getAsync(key2);
>>>>>>
>>>>>> f1 always will be complete before f2.
>>>>>>
>>>>>> I want to have the ability run multiple async operations in one
>>>>>> thread.
>>>>>> What do you think?
>>>>>>
>>>>>> Maybe we can add new cache adapter with fair async operations?
>>>>>>
>>>>>>
>>>>>>
>
Re: async operation is not fair async
Posted by Ivan Rakov <iv...@gmail.com>.
Agree. "fair" is more descriptive.
Best Regards,
Ivan Rakov
On 11.05.2018 18:30, Dmitriy Govorukhin wrote:
> Ivan,
>
> My suggestion "withFairAsync()". What do you think?
>
> On Fri, May 11, 2018 at 6:23 PM, Ivan Rakov <iv...@gmail.com> wrote:
>
>> I think, the best option from API side is to add decorating
>> withExplicitAsync() method.
>> We already have withKeepBinary, withExpiryPolicy and so on.
>>
>> Best Regards,
>> Ivan Rakov
>>
>>
>> On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
>>
>>> Vladimir,
>>>
>>> Should we create the new cache adapter or rework GridCacheAdapter?
>>>
>>> On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <vo...@gridgain.com>
>>> wrote:
>>>
>>> +1
>>>> This would also be helpful for transactional SQL as it would allow to
>>>> hide
>>>> network latency. But there is a problem - deadlocks. We need to inform
>>>> user
>>>> that this mode should be used with great care.
>>>>
>>>> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
>>>> dmitriy.govorukhin@gmail.com> wrote:
>>>>
>>>> Hi Igniters,
>>>>> I have a question. Why our async operation in not really async?
>>>>>
>>>>> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
>>>>> operation completed.
>>>>>
>>>>> This means all async operation in one thread will be executed one by one
>>>>> but not in parallel. Async operation is not async.
>>>>>
>>>>> Example for atomic cache
>>>>>
>>>>> f1=cache.getAsync(key1);
>>>>> f2=cache.getAsync(key2);
>>>>>
>>>>> f1 always will be complete before f2.
>>>>>
>>>>> I want to have the ability run multiple async operations in one thread.
>>>>> What do you think?
>>>>>
>>>>> Maybe we can add new cache adapter with fair async operations?
>>>>>
>>>>>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Ivan,
My suggestion "withFairAsync()". What do you think?
On Fri, May 11, 2018 at 6:23 PM, Ivan Rakov <iv...@gmail.com> wrote:
> I think, the best option from API side is to add decorating
> withExplicitAsync() method.
> We already have withKeepBinary, withExpiryPolicy and so on.
>
> Best Regards,
> Ivan Rakov
>
>
> On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
>
>> Vladimir,
>>
>> Should we create the new cache adapter or rework GridCacheAdapter?
>>
>> On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <vo...@gridgain.com>
>> wrote:
>>
>> +1
>>>
>>> This would also be helpful for transactional SQL as it would allow to
>>> hide
>>> network latency. But there is a problem - deadlocks. We need to inform
>>> user
>>> that this mode should be used with great care.
>>>
>>> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
>>> dmitriy.govorukhin@gmail.com> wrote:
>>>
>>> Hi Igniters,
>>>>
>>>> I have a question. Why our async operation in not really async?
>>>>
>>>> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
>>>> operation completed.
>>>>
>>>> This means all async operation in one thread will be executed one by one
>>>> but not in parallel. Async operation is not async.
>>>>
>>>> Example for atomic cache
>>>>
>>>> f1=cache.getAsync(key1);
>>>> f2=cache.getAsync(key2);
>>>>
>>>> f1 always will be complete before f2.
>>>>
>>>> I want to have the ability run multiple async operations in one thread.
>>>> What do you think?
>>>>
>>>> Maybe we can add new cache adapter with fair async operations?
>>>>
>>>>
>
Re: async operation is not fair async
Posted by Ivan Rakov <iv...@gmail.com>.
I think, the best option from API side is to add decorating
withExplicitAsync() method.
We already have withKeepBinary, withExpiryPolicy and so on.
Best Regards,
Ivan Rakov
On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
> Vladimir,
>
> Should we create the new cache adapter or rework GridCacheAdapter?
>
> On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
>> +1
>>
>> This would also be helpful for transactional SQL as it would allow to hide
>> network latency. But there is a problem - deadlocks. We need to inform user
>> that this mode should be used with great care.
>>
>> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
>> dmitriy.govorukhin@gmail.com> wrote:
>>
>>> Hi Igniters,
>>>
>>> I have a question. Why our async operation in not really async?
>>>
>>> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
>>> operation completed.
>>>
>>> This means all async operation in one thread will be executed one by one
>>> but not in parallel. Async operation is not async.
>>>
>>> Example for atomic cache
>>>
>>> f1=cache.getAsync(key1);
>>> f2=cache.getAsync(key2);
>>>
>>> f1 always will be complete before f2.
>>>
>>> I want to have the ability run multiple async operations in one thread.
>>> What do you think?
>>>
>>> Maybe we can add new cache adapter with fair async operations?
>>>
Re: async operation is not fair async
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Vladimir,
Should we create the new cache adapter or rework GridCacheAdapter?
On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> +1
>
> This would also be helpful for transactional SQL as it would allow to hide
> network latency. But there is a problem - deadlocks. We need to inform user
> that this mode should be used with great care.
>
> On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
> dmitriy.govorukhin@gmail.com> wrote:
>
> > Hi Igniters,
> >
> > I have a question. Why our async operation in not really async?
> >
> > GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
> > operation completed.
> >
> > This means all async operation in one thread will be executed one by one
> > but not in parallel. Async operation is not async.
> >
> > Example for atomic cache
> >
> > f1=cache.getAsync(key1);
> > f2=cache.getAsync(key2);
> >
> > f1 always will be complete before f2.
> >
> > I want to have the ability run multiple async operations in one thread.
> > What do you think?
> >
> > Maybe we can add new cache adapter with fair async operations?
> >
>
Re: async operation is not fair async
Posted by Vladimir Ozerov <vo...@gridgain.com>.
+1
This would also be helpful for transactional SQL as it would allow to hide
network latency. But there is a problem - deadlocks. We need to inform user
that this mode should be used with great care.
On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
dmitriy.govorukhin@gmail.com> wrote:
> Hi Igniters,
>
> I have a question. Why our async operation in not really async?
>
> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last async
> operation completed.
>
> This means all async operation in one thread will be executed one by one
> but not in parallel. Async operation is not async.
>
> Example for atomic cache
>
> f1=cache.getAsync(key1);
> f2=cache.getAsync(key2);
>
> f1 always will be complete before f2.
>
> I want to have the ability run multiple async operations in one thread.
> What do you think?
>
> Maybe we can add new cache adapter with fair async operations?
>