You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by limabean <dr...@gmail.com> on 2016/05/31 19:49:03 UTC
listening for events on transactions
Hi,
The Ignite documentation implies that listeners can be set on transactions:
"IgniteTransactions interface contains functionality for starting and
completing transactions, as well as subscribing listeners or getting
metrics."
http://apacheignite.gridgain.org/v1.6/docs/transactions#ignitetransactions
However, when I look at the interfaces/classes, it is not clear to me how to
set transaction listeners.
The behavior I am looking for is the following:
I'd like to make a call like this:
IgniteTransactionSubsystem.registerListener(myListener...)
When a transaction commits, I would like to have the listener notified
(callback) with the full contents of the transaction, even if it spanned
multiple caches....or at a minimum provide the cache keys and cache names
that were updated in a transaction.
I looked at the CacheJdbcStoreSessionListener example and saw this piece of
code in the sample persistence code:
// Configure JDBC session listener.
cacheCfg.setCacheStoreSessionListenerFactories(new
Factory<CacheStoreSessionListener>() {
@Override public CacheStoreSessionListener create() {
CacheJdbcStoreSessionListener lsnr = new
CacheJdbcStoreSessionListener();
lsnr.setDataSource(JdbcConnectionPool.create("jdbc:h2:tcp://localhost/mem:ExampleDb",
"sa", ""));
return lsnr;
}
});
but it wasn't clear to me how these were supposed to work if they were the
correct way to listen for transaction.commits.
Continuous queries appear to offer event capability, but it isn't clear how
they work with transactions.
Thanks for the clarifications/help on how to do transaction listeners.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: listening for events on transactions
Posted by limabean <dr...@gmail.com>.
Hi Alexey,
I did poke around on continuous queries before.
Based on your recommendation I will take another look to see if they solve
my architecture pattern.
Transaction listeners/messages are a common pattern with other technologies.
Here is a suggestion of how Ignite might evolve to be better in this area:
I envision an Ignite server in the grid acting as a transaction coordinator.
(I have Cassandra behavior in mind when thinking through this).
The client code may issue the final "commit" API call, but this would simply
send an indicator to the coordinator in the grid which would handle the
commit
across the grid with the data owners. Then, this coordinator could generate
a transaction message to any registered listeners AND return the transaction
status
back to the client.
Guessing about how Ignite works today, there appears to be an assumption
that the client...even the end user client code...has to be the coordinator
of
the transaction. But I don't think it has to remain this way. A model more
similar to Cassandra behavior (I'm not talking Cassandra transactions - it
only
has lightweight, query specific transactions - but more the model of the way
a client interacts with the Cassandras cluster), might be an interesting
model
for Ignite to evolve towards.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5460.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: listening for events on transactions
Posted by Alexey Goncharuk <al...@gmail.com>.
David,
Have you considered using continuous queries for your use-case [1]? Even if
there were such a thing as a transaction event, I do not see how you can
reliably (read - in a proper order) publish this information to Kafka.
Say, you have 2 clients and 2 server nodes. First client executes a
transaction on keys (1, 2) and second client executes a transaction on keys
(2, 3). These events would have been fired on client nodes because only
client knows when the whole transaction is committed on all nodes. Now you
get two unordered events (1, 2) and (2, 3) which gives you an undefined
state for key (2), or you need to add some sort of custom logic to order
these updates.
The solution would be to listen for updates on primary nodes, which is
exactly what continuous queries do. They guarantee that:
- Updates for each key will be received in a proper order
- Events are fault-tolerant, meaning that even if primary node crashed,
you would still receive an update from backup node
[1] https://apacheignite.readme.io/docs/continuous-queries
2016-06-01 13:16 GMT-07:00 Denis Magda <dm...@gridgain.com>:
> Hi David,
>
> I would implement this using custom events in the following way:
>
> - introduce your custom event - TransactionEvent and send it every time to
> subscribers a transaction succeeds using IgniteEvents.recordLocal(…)
> method. The event can contain any info you need.
>
> - the separate process can subscribe for your custom TransactionEvents,
> process them and publish to Kafka.
>
> This will let you to decouple Ignite logic from Kafka logic.
>
> Below is a code snippet that demonstrates how to use custom events in
> general:
> https://gist.github.com/dmagda/f1b93a9aa4774a360465bc9f63cc58c9
>
> Will this work for you?
>
> —
> Denis
>
> On Jun 1, 2016, at 9:09 PM, David Robinson <dr...@gmail.com> wrote:
>
> I can - but apparently not with a separate process that gets notifications
> - which is the design point I was seeking.
> Thanks again.
>
> On Wed, Jun 1, 2016 at 12:55 PM, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
>> You can easily implement such a thing as a part of transaction logic.
>>
>>
>> 2016-06-01 14:52 GMT+03:00 limabean <dr...@gmail.com>:
>>
>>> Hi Alexi,
>>>
>>> Thank you for the clarification.
>>>
>>> My final goal is to notify other processes not related to the grid
>>> application about changes to the data caches. For example, it would be
>>> nice
>>> to have a Kafka publisher registered as a transaction listener and then
>>> when
>>> it gets transaction events, publish this information to a Kafka topic.
>>>
>>> Ignite is good at pulling data in, but it needs to be equally good at
>>> sharing data to work well in my environment.
>>>
>>> What do you think ?
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com
>>> <http://nabble.com>.
>>>
>>
>>
>>
>> --
>>
>> Best regards,
>> Alexei Scherbakov
>>
>
>
>
Re: listening for events on transactions
Posted by limabean <dr...@gmail.com>.
Hey Denis,
Thank you for this suggestion. I plan to take a serious look at what you
suggest to
see if it will work for me and will let you know.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5458.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: listening for events on transactions
Posted by Denis Magda <dm...@gridgain.com>.
Hi David,
I would implement this using custom events in the following way:
- introduce your custom event - TransactionEvent and send it every time to subscribers a transaction succeeds using IgniteEvents.recordLocal(…) method. The event can contain any info you need.
- the separate process can subscribe for your custom TransactionEvents, process them and publish to Kafka.
This will let you to decouple Ignite logic from Kafka logic.
Below is a code snippet that demonstrates how to use custom events in general:
https://gist.github.com/dmagda/f1b93a9aa4774a360465bc9f63cc58c9
Will this work for you?
—
Denis
> On Jun 1, 2016, at 9:09 PM, David Robinson <dr...@gmail.com> wrote:
>
> I can - but apparently not with a separate process that gets notifications - which is the design point I was seeking.
> Thanks again.
>
> On Wed, Jun 1, 2016 at 12:55 PM, Alexei Scherbakov <alexey.scherbakoff@gmail.com <ma...@gmail.com>> wrote:
> You can easily implement such a thing as a part of transaction logic.
>
>
> 2016-06-01 14:52 GMT+03:00 limabean <drobin1437@gmail.com <ma...@gmail.com>>:
> Hi Alexi,
>
> Thank you for the clarification.
>
> My final goal is to notify other processes not related to the grid
> application about changes to the data caches. For example, it would be nice
> to have a Kafka publisher registered as a transaction listener and then when
> it gets transaction events, publish this information to a Kafka topic.
>
> Ignite is good at pulling data in, but it needs to be equally good at
> sharing data to work well in my environment.
>
> What do you think ?
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html <http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html>
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
Re: listening for events on transactions
Posted by David Robinson <dr...@gmail.com>.
I can - but apparently not with a separate process that gets notifications
- which is the design point I was seeking.
Thanks again.
On Wed, Jun 1, 2016 at 12:55 PM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:
> You can easily implement such a thing as a part of transaction logic.
>
>
> 2016-06-01 14:52 GMT+03:00 limabean <dr...@gmail.com>:
>
>> Hi Alexi,
>>
>> Thank you for the clarification.
>>
>> My final goal is to notify other processes not related to the grid
>> application about changes to the data caches. For example, it would be
>> nice
>> to have a Kafka publisher registered as a transaction listener and then
>> when
>> it gets transaction events, publish this information to a Kafka topic.
>>
>> Ignite is good at pulling data in, but it needs to be equally good at
>> sharing data to work well in my environment.
>>
>> What do you think ?
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
Re: listening for events on transactions
Posted by Alexei Scherbakov <al...@gmail.com>.
You can easily implement such a thing as a part of transaction logic.
2016-06-01 14:52 GMT+03:00 limabean <dr...@gmail.com>:
> Hi Alexi,
>
> Thank you for the clarification.
>
> My final goal is to notify other processes not related to the grid
> application about changes to the data caches. For example, it would be
> nice
> to have a Kafka publisher registered as a transaction listener and then
> when
> it gets transaction events, publish this information to a Kafka topic.
>
> Ignite is good at pulling data in, but it needs to be equally good at
> sharing data to work well in my environment.
>
> What do you think ?
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexei Scherbakov
Re: listening for events on transactions
Posted by limabean <dr...@gmail.com>.
Hi Alexi,
Thank you for the clarification.
My final goal is to notify other processes not related to the grid
application about changes to the data caches. For example, it would be nice
to have a Kafka publisher registered as a transaction listener and then when
it gets transaction events, publish this information to a Kafka topic.
Ignite is good at pulling data in, but it needs to be equally good at
sharing data to work well in my environment.
What do you think ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346p5357.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: listening for events on transactions
Posted by Alexei Scherbakov <al...@gmail.com>.
Hi,
I think the documentation is not very precise on tx topic.
Currently where is no way to listen for transaction events.
The only exception is CacheStoreSessionListener interface which works in
conjunction with CacheStore.
I suppose you can implement "fake" CacheStore and track transaction
activity in corresponding methods.
What's your final goal?
2016-05-31 22:49 GMT+03:00 limabean <dr...@gmail.com>:
> Hi,
>
> The Ignite documentation implies that listeners can be set on transactions:
>
> "IgniteTransactions interface contains functionality for starting and
> completing transactions, as well as subscribing listeners or getting
> metrics."
> http://apacheignite.gridgain.org/v1.6/docs/transactions#ignitetransactions
>
>
> However, when I look at the interfaces/classes, it is not clear to me how
> to
> set transaction listeners.
> The behavior I am looking for is the following:
>
> I'd like to make a call like this:
> IgniteTransactionSubsystem.registerListener(myListener...)
>
> When a transaction commits, I would like to have the listener notified
> (callback) with the full contents of the transaction, even if it spanned
> multiple caches....or at a minimum provide the cache keys and cache names
> that were updated in a transaction.
>
> I looked at the CacheJdbcStoreSessionListener example and saw this piece of
> code in the sample persistence code:
> // Configure JDBC session listener.
> cacheCfg.setCacheStoreSessionListenerFactories(new
> Factory<CacheStoreSessionListener>() {
> @Override public CacheStoreSessionListener create() {
> CacheJdbcStoreSessionListener lsnr = new
> CacheJdbcStoreSessionListener();
>
>
>
> lsnr.setDataSource(JdbcConnectionPool.create("jdbc:h2:tcp://localhost/mem:ExampleDb",
> "sa", ""));
>
> return lsnr;
> }
> });
>
> but it wasn't clear to me how these were supposed to work if they were the
> correct way to listen for transaction.commits.
>
> Continuous queries appear to offer event capability, but it isn't clear how
> they work with transactions.
>
> Thanks for the clarifications/help on how to do transaction listeners.
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/listening-for-events-on-transactions-tp5346.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexei Scherbakov