Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Hi is anyone able to help look into/reproduce this with the example code
given? thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

Unfortunately, I'm not aware of it.

Please note that Ignite is strongly consistent so these B's should arrive,
barring their subsequent modification or some other error.

Regards,
-- 
Ilya Kasnacheev


чт, 17 сент. 2020 г. в 17:22, ssansoy <s....@cmcmarkets.com>:

> Yeah sadly that is the only option we can think of too.... Not very nice
> though - and we'd still need to somehow transmit information in the
> callback
> about how many B's to wait for - and what to do if they never arrive etc
>
> Is there some other blocking operation on the B cache perhaps we can call
> to
> act as a barrier, after which we can be sure all the B's have arrived?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Yeah sadly that is the only option we can think of too.... Not very nice
though - and we'd still need to somehow transmit information in the callback
about how many B's to wait for - and what to do if they never arrive etc

Is there some other blocking operation on the B cache perhaps we can call to
act as a barrier, after which we can be sure all the B's have arrived?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

You can repeat SQL query until you see all of B.

Regards,
-- 
Ilya Kasnacheev


чт, 17 сент. 2020 г. в 13:15, ssansoy <s....@cmcmarkets.com>:

> Hi, thanks - that is broadly what we are doing:
>
> 1. A continuous query on A (annotated with @IgniteAsyncCallback).
> 2. When an update comes in, fetch all the associated keys of B.
>
> We were doing step 2 with an SQL fields query - and the issue was that not
> all the B's were available yet because the transaction that updates A and
> Bs
> isn't finished, yet step 1 occurs anyway.
>
> It sounds like we want to use a IgniteCache operation rather than an SQL
> fields query to get B (e.g. assuming cache.get(keys) will block until the
> B's are available). However, the issue is that we can't do a cache.get
> using
> the fields from A. E.g. for a single A there are many B's so we do not have
> the set of primary keys of B at this point.
> I dont think we can even do a getAll and then filter locally as there is no
> getAll in the api that doesn't require a key.
>
> Any other suggestions?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Hi, thanks - that is broadly what we are doing:

1. A continuous query on A (annotated with @IgniteAsyncCallback).
2. When an update comes in, fetch all the associated keys of B.

We were doing step 2 with an SQL fields query - and the issue was that not
all the B's were available yet because the transaction that updates A and Bs
isn't finished, yet step 1 occurs anyway.

It sounds like we want to use a IgniteCache operation rather than an SQL
fields query to get B (e.g. assuming cache.get(keys) will block until the
B's are available). However, the issue is that we can't do a cache.get using
the fields from A. E.g. for a single A there are many B's so we do not have
the set of primary keys of B at this point.
I dont think we can even do a getAll and then filter locally as there is no
getAll in the api that doesn't require a key.

Any other suggestions?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

I think you can trigger an async callback when A or any of B is updated.
The callback will process these entries: it will wait some reasonable time
if some entry is still "old". You can use a timestamp on every entry for
that.

If there's a second update on A, you may skip some callbacks.

Regards,
-- 
Ilya Kasnacheev


ср, 16 сент. 2020 г. в 17:57, ssansoy <s....@cmcmarkets.com>:

> Are there any other ways we can model this to make this problem easier to
> solve with Ignite (95% of our other caches don't have this requirement, but
> we need to solve the 5% to migrate to ignite from our legacy solution)
>
> The original thread on this is here btw:
>
>
> http://apache-ignite-users.70518.x6.nabble.com/Parent-Child-relationships-td31605.html
>
> E.g. Parent A, Child B.
> A contains a list of B's
> If A's fields, or any of it's B's fields are updated, then the client's
> callback is triggered with A and all the B's so the entire structure can be
> processed.
>
> A and B need to be editable via SQL by human users e.g. be represented as
> table A and table B (one of the main benefits Ignite provides over our
> current legacy system).
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Are there any other ways we can model this to make this problem easier to
solve with Ignite (95% of our other caches don't have this requirement, but
we need to solve the 5% to migrate to ignite from our legacy solution)

The original thread on this is here btw:

http://apache-ignite-users.70518.x6.nabble.com/Parent-Child-relationships-td31605.html

E.g. Parent A, Child B.
A contains a list of B's
If A's fields, or any of it's B's fields are updated, then the client's
callback is triggered with A and all the B's so the entire structure can be
processed.

A and B need to be editable via SQL by human users e.g. be represented as
table A and table B (one of the main benefits Ignite provides over our
current legacy system).



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

I think this suggests data model adjustment.

Regards,
-- 
Ilya Kasnacheev


вт, 15 сент. 2020 г. в 17:46, Evgenii Zhuravlev <e....@gmail.com>:

> Ilya,
>
> This won't help, since the problem here is that CQ doesn't return all
> needed keys.
>
> Evgenii
>
> вт, 15 сент. 2020 г. в 02:28, Ilya Kasnacheev <il...@gmail.com>:
>
>> Hello!
>>
>> Maybe keys may be queued from the CQ to be revisited later with
>> transaction per key approach.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 14 сент. 2020 г. в 21:15, Evgenii Zhuravlev <e.zhuravlev.wk@gmail.com
>> >:
>>
>>> No, I don't see other ways to do this transactionally, as CQ itself is
>>> not transactional.
>>>
>>> Evgenii
>>>
>>> чт, 10 сент. 2020 г. в 00:52, ssansoy <s....@cmcmarkets.com>:
>>>
>>>> unfortunately the 's' on B here can't be derived from a number 0..n -
>>>> e.g. it
>>>> isn't a numeric id.
>>>>
>>>> E.g. in practice lets say:
>>>>
>>>> A is a "Location"
>>>> it has properties: "city", "street" etc
>>>>
>>>> B is a "Person" with key:
>>>> p = city
>>>> q = street
>>>> r = social security number
>>>>
>>>> E.g. an A and associated B's are updated in a transaction, we want our
>>>> client app to see the updated A and B's where the Person lives at that
>>>> that
>>>> Location.
>>>>
>>>> E.g. A is updated and our continuous query on A picks up:
>>>> city = London
>>>> street = Downing Street
>>>>
>>>> We would like to say:
>>>> Select * from B where city="London" and street="Downing Street"
>>>>
>>>> Is there any way at all in Ignite to do this transactionally, so if an
>>>> A and
>>>> associated B's are updated in one transaction (e.g. a street is renamed
>>>> from
>>>> "Downing Street" to "Regent Street"), then our client app can see them
>>>> consistently?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

Ilya,

This won't help, since the problem here is that CQ doesn't return all
needed keys.

Evgenii

вт, 15 сент. 2020 г. в 02:28, Ilya Kasnacheev <il...@gmail.com>:

> Hello!
>
> Maybe keys may be queued from the CQ to be revisited later with
> transaction per key approach.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 14 сент. 2020 г. в 21:15, Evgenii Zhuravlev <e.zhuravlev.wk@gmail.com
> >:
>
>> No, I don't see other ways to do this transactionally, as CQ itself is
>> not transactional.
>>
>> Evgenii
>>
>> чт, 10 сент. 2020 г. в 00:52, ssansoy <s....@cmcmarkets.com>:
>>
>>> unfortunately the 's' on B here can't be derived from a number 0..n -
>>> e.g. it
>>> isn't a numeric id.
>>>
>>> E.g. in practice lets say:
>>>
>>> A is a "Location"
>>> it has properties: "city", "street" etc
>>>
>>> B is a "Person" with key:
>>> p = city
>>> q = street
>>> r = social security number
>>>
>>> E.g. an A and associated B's are updated in a transaction, we want our
>>> client app to see the updated A and B's where the Person lives at that
>>> that
>>> Location.
>>>
>>> E.g. A is updated and our continuous query on A picks up:
>>> city = London
>>> street = Downing Street
>>>
>>> We would like to say:
>>> Select * from B where city="London" and street="Downing Street"
>>>
>>> Is there any way at all in Ignite to do this transactionally, so if an A
>>> and
>>> associated B's are updated in one transaction (e.g. a street is renamed
>>> from
>>> "Downing Street" to "Regent Street"), then our client app can see them
>>> consistently?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: Lag before records are visible after transaction commit

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

Maybe keys may be queued from the CQ to be revisited later with transaction
per key approach.

Regards,
-- 
Ilya Kasnacheev


пн, 14 сент. 2020 г. в 21:15, Evgenii Zhuravlev <e....@gmail.com>:

> No, I don't see other ways to do this transactionally, as CQ itself is not
> transactional.
>
> Evgenii
>
> чт, 10 сент. 2020 г. в 00:52, ssansoy <s....@cmcmarkets.com>:
>
>> unfortunately the 's' on B here can't be derived from a number 0..n -
>> e.g. it
>> isn't a numeric id.
>>
>> E.g. in practice lets say:
>>
>> A is a "Location"
>> it has properties: "city", "street" etc
>>
>> B is a "Person" with key:
>> p = city
>> q = street
>> r = social security number
>>
>> E.g. an A and associated B's are updated in a transaction, we want our
>> client app to see the updated A and B's where the Person lives at that
>> that
>> Location.
>>
>> E.g. A is updated and our continuous query on A picks up:
>> city = London
>> street = Downing Street
>>
>> We would like to say:
>> Select * from B where city="London" and street="Downing Street"
>>
>> Is there any way at all in Ignite to do this transactionally, so if an A
>> and
>> associated B's are updated in one transaction (e.g. a street is renamed
>> from
>> "Downing Street" to "Regent Street"), then our client app can see them
>> consistently?
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

No, I don't see other ways to do this transactionally, as CQ itself is not
transactional.

Evgenii

чт, 10 сент. 2020 г. в 00:52, ssansoy <s....@cmcmarkets.com>:

> unfortunately the 's' on B here can't be derived from a number 0..n - e.g.
> it
> isn't a numeric id.
>
> E.g. in practice lets say:
>
> A is a "Location"
> it has properties: "city", "street" etc
>
> B is a "Person" with key:
> p = city
> q = street
> r = social security number
>
> E.g. an A and associated B's are updated in a transaction, we want our
> client app to see the updated A and B's where the Person lives at that that
> Location.
>
> E.g. A is updated and our continuous query on A picks up:
> city = London
> street = Downing Street
>
> We would like to say:
> Select * from B where city="London" and street="Downing Street"
>
> Is there any way at all in Ignite to do this transactionally, so if an A
> and
> associated B's are updated in one transaction (e.g. a street is renamed
> from
> "Downing Street" to "Regent Street"), then our client app can see them
> consistently?
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

unfortunately the 's' on B here can't be derived from a number 0..n - e.g. it
isn't a numeric id.

E.g. in practice lets say:

A is a "Location"
it has properties: "city", "street" etc

B is a "Person" with key:
p = city
q = street
r = social security number

E.g. an A and associated B's are updated in a transaction, we want our
client app to see the updated A and B's where the Person lives at that that
Location.

E.g. A is updated and our continuous query on A picks up:
city = London
street = Downing Street

We would like to say:
Select * from B where city="London" and street="Downing Street"

Is there any way at all in Ignite to do this transactionally, so if an A and
associated B's are updated in one transaction (e.g. a street is renamed from
"Downing Street" to "Regent Street"), then our client app can see them
consistently?







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

Yes, but if you know the number of entries B for this object A, then you
can get all objects using s, which will be 0..n

Evgenii

пн, 7 сент. 2020 г. в 06:38, ssansoy <s....@cmcmarkets.com>:

> Thanks Evgenii,
>
> Sorry to keep revisiting this - maybe I am misunderstanding, but don't we
> also need 's' to be able to query B by key. E.g. the key of B consists of
> {q, r, s} We only have q and r from the parent A.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Thanks Evgenii,

Sorry to keep revisiting this - maybe I am misunderstanding, but don't we
also need 's' to be able to query B by key. E.g. the key of B consists of
{q, r, s} We only have q and r from the parent A.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

You can put the number of entries in B cache related to this object A right
in the object A. After that, you can use this number to make keys of all
objects from cache B, as you already know q and r. But it depends on use
case.

Evgenii

пт, 4 сент. 2020 г. в 03:21, ssansoy <s....@cmcmarkets.com>:

> Thanks Evgenii,
>
> Could you please elaborate a bit on how the get would work here.
>
> E.g. parent object A has properties p, q, r
> child object B has properties q, r, s, t
>
> {q, r, s} are the primary key of B (as defined in backing SQL table DDL
> which is how the cache was created)
>
> When an A update comes in with values p1, q1, r1, we were doing a select *
> from B where q=q1 and r=r1 which would return multiple records.
>
> Is there an equivalent using igniteCacheForB.get(key). What would key be
> here?
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Thanks Evgenii,

Could you please elaborate a bit on how the get would work here.

E.g. parent object A has properties p, q, r
child object B has properties q, r, s, t

{q, r, s} are the primary key of B (as defined in backing SQL table DDL
which is how the cache was created)

When an A update comes in with values p1, q1, r1, we were doing a select *
from B where q=q1 and r=r1 which would return multiple records.

Is there an equivalent using igniteCacheForB.get(key). What would key be
here?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

Yes, it is expected that ScanQuery and ContinuousQuery are not
transactional.

>Getting all the records will pull everything back onto the and we would
have to filter locally if I am understanding correctly?
There is no need to get all entries from the cache, you can get entries
with certain keys. In your example, you can get all entries based on
generated keys if you know the number of inserted entries. This number, for
example, can be inserted as a part of the first object.

Evgenii

ср, 2 сент. 2020 г. в 08:53, ssansoy <s....@cmcmarkets.com>:

> Thanks for looking into it. Is this expected?
> Just wondering how another node can ever be transactionally notified of an
> update in an event driven way if continuous queries don't support
> transactions?
>
> using getAll isn't a practical workaround unfortunately as we want to get
> the B records based on the value of some of it's fields. E.g. a scan query
> with a filter on the child, or an sql fields query. Getting all the records
> will pull everything back onto the and we would have to filter locally if I
> am understanding correctly?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Lag before records are visible after transaction commit

[ssansoy <s....@cmcmarkets.com>]

Thanks for looking into it. Is this expected?
Just wondering how another node can ever be transactionally notified of an
update in an event driven way if continuous queries don't support
transactions?

using getAll isn't a practical workaround unfortunately as we want to get
the B records based on the value of some of it's fields. E.g. a scan query
with a filter on the child, or an sql fields query. Getting all the records
will pull everything back onto the and we would have to filter locally if I
am understanding correctly?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

Hi,

To make this work, you can change the transaction type - from READ_COMMITED
to SERIALIZABLE and replace scanQuery with getAll. In this case, getAll
operation will be waiting for locked keys. Note that running cache
operations in CQ listener thread may cause deadlock and it's better to use
another thread for that.

Best Regards,
Evgenii

Re: Lag before records are visible after transaction commit

[VeenaMithare <v....@cmcmarkets.com>]

Hi , 

>>Continuous Query itself is not transactional and it looks like it can't be
used for this at the moment. So, it gets notification before other entries
were committed.

Does this mean we could get dirty reads as updates in continuous query ?
i.e. for eg if the code is as below:
1. Start transaction  
2. update records of cache a
3. update records of cache b 
4. update records for cache c
5. commit

if update of cache a succeeds , but update of cache b fails, will the local
listener for continuous query for 'cache a' get an update ?

regards,
Veena.


regards
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[Evgenii Zhuravlev <e....@gmail.com>]

Hi,

Checked this reproducer. Continuous Query itself is not transactional and
it looks like it can't be used for this at the moment. So, it gets
notification before other entries were committed.

Best Regards,
Evgenii

вт, 1 сент. 2020 г. в 00:34, ssansoy <s....@cmcmarkets.com>:

> Hi is anyone able to help look into/reproduce this with the example code
> given? thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>