You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "steve.hostettler" <st...@gmail.com> on 2017/04/25 16:12:24 UTC
Write behind and eventual consistency
Hello,
Iwould like to enable the write-behind mode but since Ihave to sync the
overall process with other jobs (extract to the datawarehouse), I would like
to know how to make sure that there are no more write behind operations
waiting for completion. I had a look at the API but I do not see anything
like that.
What would you advise to do?
Many thanks in advance
Best Regards
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Write behind and eventual consistency
Posted by vkulichenko <va...@gmail.com>.
If it's this kind of batch processing, and if you're still waiting for a
batch to be processed completely before the next one can be processed, why
do you want to use write-behind? What is wrong with sync write-through in
this case?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12438.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Write behind and eventual consistency
Posted by Gaurav Bajaj <ga...@gmail.com>.
Any updates on the two use cases mentioned above?
On Apr 27, 2017 10:53 AM, "Gaurav Bajaj" <ga...@gmail.com> wrote:
> Hi Val,
>
> Our use case :
>
> 1. Read records from file
> 2. Do computations on each record
> 3. Put them in the cache and persistence using write behind.
> 4. When all the records from file are processed, updated in Cache and also
> persisted to DB, we want to
> trigger some other process which will do next set of operations on these
> records from cache.
> 5. We want to trigger this next process and mark original file as
> processed, only when we are sure data is persisted, so that in case of Node
> failure we need not process that file again.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Apr 27, 2017 at 10:26 AM, steve.hostettler <
> steve.hostettler@gmail.com> wrote:
>
>> Hi Val,
>>
>> the use case is the following
>>
>> 1) Load data into the database from an external system
>> 2) Once ready load it into the grid
>> 3) Process something that does massive write behinds
>> 4) Take a snapshot of the results (or) Do a backup of the tables <<---
>> At
>> this point I need the eventual consistency to ...eventually be
>>
>> At step 4 I cannot afford to have some update still in progress. This is
>> even more important since because of write behind I cannot maintain
>> referential integrity (since the insert/update are done in a random order)
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12287.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: Write behind and eventual consistency
Posted by Gaurav Bajaj <ga...@gmail.com>.
Hi Val,
Our use case :
1. Read records from file
2. Do computations on each record
3. Put them in the cache and persistence using write behind.
4. When all the records from file are processed, updated in Cache and also
persisted to DB, we want to
trigger some other process which will do next set of operations on these
records from cache.
5. We want to trigger this next process and mark original file as
processed, only when we are sure data is persisted, so that in case of Node
failure we need not process that file again.
On Thu, Apr 27, 2017 at 10:26 AM, steve.hostettler <
steve.hostettler@gmail.com> wrote:
> Hi Val,
>
> the use case is the following
>
> 1) Load data into the database from an external system
> 2) Once ready load it into the grid
> 3) Process something that does massive write behinds
> 4) Take a snapshot of the results (or) Do a backup of the tables <<--- At
> this point I need the eventual consistency to ...eventually be
>
> At step 4 I cannot afford to have some update still in progress. This is
> even more important since because of write behind I cannot maintain
> referential integrity (since the insert/update are done in a random order)
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Write-behind-and-eventual-
> consistency-tp12242p12287.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
RE: Write behind and eventual consistency
Posted by "steve.hostettler" <st...@gmail.com>.
Hi Val,
the use case is the following
1) Load data into the database from an external system
2) Once ready load it into the grid
3) Process something that does massive write behinds
4) Take a snapshot of the results (or) Do a backup of the tables <<--- At
this point I need the eventual consistency to ...eventually be
At step 4 I cannot afford to have some update still in progress. This is
even more important since because of write behind I cannot maintain
referential integrity (since the insert/update are done in a random order)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12287.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
RE: Write behind and eventual consistency
Posted by vkulichenko <va...@gmail.com>.
Hi Steve,
What is the business use case behind this?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12285.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
RE: Write behind and eventual consistency
Posted by Steve Hostettler <st...@gmail.com>.
Hello Val,
Thanks for your help. Don't you think that this would be an interesting
functionality?
Steve
-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Wednesday, April 26, 2017 9:57 AM
To: user@ignite.apache.org
Subject: Re: Write behind and eventual consistency
There is no way to do this. However, you can take a look at
GridCacheWriteBehindStore which implements this functionality and try
tweaking it so that the queue size is exposed somehow.
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-con
sistency-tp12242p12256.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Write behind and eventual consistency
Posted by vkulichenko <va...@gmail.com>.
There is no way to do this. However, you can take a look at
GridCacheWriteBehindStore which implements this functionality and try
tweaking it so that the queue size is exposed somehow.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12256.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Write behind and eventual consistency
Posted by vkulichenko <va...@gmail.com>.
Hi Steve,
I don't think it's currently possible and frankly I'm not sure I understand
what it actually means. Can you clarify what is implied in "no more write
behind operations waiting for completion"? We could probably check if the
queue is empty, but what if new updates happen right after or concurrently
with this check?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12247.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.