You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by zaleslaw <za...@gmail.com> on 2020/08/03 16:27:10 UTC

Re: Why does CacheBasedDataSet destroy the cache it is given

Dear Courtney Robinson, let's discuss here the possible behaviour of this
CacheBased Dataset closing.

When designed this feature we think, that the all training parts and stuff
should be deleted from Caches ad model should be serialized or exported
somwhere.

What is your use-case& Could you share some code or pseudo-code?
How are you going to handle data after training?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Why does CacheBasedDataSet destroy the cache it is given

Posted by zaleslaw <za...@gmail.com>.
Dear Courtney Robinson, 

please write if you have any cases to update helper cache with paritions to
better understand the situation. 

How and when are you going to clear this helper caches if the alternative
version (as you suggested in the first email) of CacheBased Dataset will be
provided?

Also, could you please answer to akornesh about main data cache behaviour?
Is it really destroyed in your tests or not now?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Why does CacheBasedDataSet destroy the cache it is given

Posted by akorensh <al...@gmail.com>.
Courtney,

 The CacheBasedDataset.close() method below only destroys the helper cache
derived from 
  the original data, and used to train the model. It does not touch the
original data set.


@Override public void close() {
    datasetCache.destroy(); // destroy the helper cache derived from the
original cache
    ComputeUtils.removeData(ignite, datasetId); // remove helper data stored
locally on a node.
    ComputeUtils.removeLearningEnv(ignite, datasetId); //remove helper
object used to make the model.
}

see:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/dataset/CacheBasedDatasetExample.java

If you follow the above example, remove the persons.destroy() statement,
remove Ignite from the auto close block,  run it, and connect via web
console, you would see that the original persons data set remains intact.

If for some reason you do need the helper cache that was created to train
the model then do as follows: 1. create your own: MyCacheBasedDataSet
extends CacheBaseDataSet 
2. override the close() method. This is not recommended for prod, but could
be useful for debugging the models. 


Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Why does CacheBasedDataSet destroy the cache it is given

Posted by Courtney Robinson <co...@hypi.io>.
Hey,
Just seen this reply.
We have Ignite persistence enabled. The caches/tables are the primary
source of the data. That's the use case.
If we build an ML model from the data in a cache, Ignite's behaviour of
deleting the cache means we'll have lost that data.
We were just lucky this showed up in tests before it got anywhere near
production data.

In our case, we're push data into a cache continually and rebuilding the
model periodically.

Regards,
Courtney Robinson
Founder and CEO, Hypi
Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io>

<https://hypi.io>
https://hypi.io


On Mon, Aug 3, 2020 at 5:28 PM zaleslaw <za...@gmail.com> wrote:

> Dear Courtney Robinson, let's discuss here the possible behaviour of this
> CacheBased Dataset closing.
>
> When designed this feature we think, that the all training parts and stuff
> should be deleted from Caches ad model should be serialized or exported
> somwhere.
>
> What is your use-case& Could you share some code or pseudo-code?
> How are you going to handle data after training?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>