Ignite persistence: Data of node lost that got excluded from baseline topology

[VincentCE <ve...@cephei.com>]

Hi!

in our project we are currently using ignite 2.8.1 without ignite native
persistence enabled. No we would like to enable this feature to prevent data
loss during node restart. 

Important: We use AttributeNodeFilters to separate our data, e.g data of
type1 only lives in the type1-clustergroup and so on.

I have two question regarding *native persistence*:

1. After some time we would like to shut down nodes of type1 to save
resources (but possibly would like to use it in the future again if business
requires it -> we would like the data to remain there safely/persisted).
However in order to start up the cluster again the type1 nodes need to be
excluded from baseline topology. But then if later on we would like to reuse
type1 nodes again their data is getting deleted as soon as they are
rejoining the baseline topology. Is there a way to prevent this?

2. When using AttributeNodeFilter it seems that we need to use fixed
consistentIds (e.g. consistentId = "type1-0") since otherwise a node of
type2 would potentially use the directory of a type1 node in the persistence
storage directory which finally would break the data separation when new
data is loaded into the caches.

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite persistence: Data of node is lost after being excluded from baseline topology

[VincentCE <ve...@cephei.com>]

Hi igniters!

could I have short answers to my questions?

Thanks a lot!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite persistence: Data of node lost that got excluded from baseline topology

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

1. I guess it's possible to activate a cluster manually when not all
baseline nodes are in. Later you can join your node back to baseline,
hopefully. Please note that there's still chance that old data will be
deleted from node upon join: We can only re-use data from old partition
when we're confident that it's identical to new data of same partition
(inactive cluster) or when we can apply historical rebalancing (apply
changes from WAL to the same partition).Historical rebalance is usually
only feasible when there's few differences between old and new partition.
If the difference is significant, it's easier to rebalance the whole
partition.

It may seem strange that we throw a lot of data out, but we don't have any
obvious way to tell which of that data is current and which is not. So it's
useless in the end.

2. Yes, it is recommended to always have consistentId in persistent
cluster. Otherwise the node will just assume first available data dir as
its own.

Regards,
-- 
Ilya Kasnacheev


пн, 23 нояб. 2020 г. в 11:17, VincentCE <ve...@cephei.com>:

> Hi!
>
> in our project we are currently using ignite 2.8.1 without ignite native
> persistence enabled. No we would like to enable this feature to prevent
> data
> loss during node restart.
>
> Important: We use AttributeNodeFilters to separate our data, e.g data of
> type1 only lives in the type1-clustergroup and so on.
>
> I have two question regarding *native persistence*:
>
> 1. After some time we would like to shut down nodes of type1 to save
> resources (but possibly would like to use it in the future again if
> business
> requires it -> we would like the data to remain there safely/persisted).
> However in order to start up the cluster again the type1 nodes need to be
> excluded from baseline topology. But then if later on we would like to
> reuse
> type1 nodes again their data is getting deleted as soon as they are
> rejoining the baseline topology. Is there a way to prevent this?
>
> 2. When using AttributeNodeFilter it seems that we need to use fixed
> consistentIds (e.g. consistentId = "type1-0") since otherwise a node of
> type2 would potentially use the directory of a type1 node in the
> persistence
> storage directory which finally would break the data separation when new
> data is loaded into the caches.
>
> Thanks in advance!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>