Embedded ignite and baseline upgrade questions

[do...@gmail.com]

Hi,

 

I am implementing distributed crawler engine using ignite in embedded mode.
Each crawler node have persistence enabled and assigned predefined
consistentId (dev-1, dev-2, dev-3, etc). Baseline topology (nodes which
store data) can change and configured using config file like this:

 

# Cluster settings

cluster {

    # Baseline topology (list of node ids which store data)

    baseline = [

        "dev-1",

        "dev-2",

        "dev-3"

    ]

}

 

Each crawler node startup implemented like this:

 

        // Start ignite

        ignite = Ignition.start(igniteConfig)

 

        // Activate

        ignite.cluster().active(true)

 

        // Validate baseline topology

        val baseline =
config.getStringAll(SpiderNodeOptions.CLUSTER_BASELINE).toSet()

        val curTopology = ignite.cluster().currentBaselineTopology()

        if (curTopology == null || baseline != curTopology.map {
it.consistentId().toString() }.toSet()) {

            // Wait for all new topology members to join

            log.info("Cluster baseline topology changed - waiting data nodes
to join...")

            while (true) {

                val nodes = mutableListOf<ClusterNode>()

                ignite.cluster().nodes().forEach {

                    if (baseline.contains(it.consistentId().toString()))

                        nodes.add(it)

                }

 

                if (nodes.size == baseline.size) {

                    log.info("Updated baseline topology of {} nodes",
value("baseline_size", nodes.size))

                    ignite.cluster().setBaselineTopology(nodes)

                    break

                }

                else

                    Thread.sleep(1000)

            }

        }

 

We can't set baseline topology before cluster activation so activate()
called first. Is it possible to set baseline topology without activating
cluster for first time ? Is it safe to activate node with the code above ?
It seems that we have misconfigured state in clean cluster between
activate() and waiting nodes to join when setBaseliteTopology not called
yet. The steps for upgrading topology will be 1) modify and commit config
files 2) shutdown all cluster members 3) update code 4) start each node one
by one. Is this the right way of reconfiguration ?

 

Thanks,

Michael

Re: Embedded ignite and baseline upgrade questions

[djm132 <do...@gmail.com>]

Thanks, its definitely clear now that rebalancing should be triggered from
code if node removal detected. Assuming that number of backups > 0 and only
one node removed it looks like safe case. But what if backup count = 0 (bad
idea but the risk may be acceptable in some cases) and we need to shutdown
node to remove it from BL ? Is there any way to trigger exclusive
rebalancing (ie moving out all data from some node to other valid nodes) and
than exclude this node from data storage ? 

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Embedded ignite and baseline upgrade questions

[akorensh <al...@gmail.com>]

Hi,
  Your summary looks correct.
  It should be possible to manage your baseline topology using a config
file, provided you follow the steps outlined.

more info here:
https://apacheignite.readme.io/docs/baseline-topology#section-triggering-rebalancing-programmatically

  You can use
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/events/CacheRebalancingEvent.html
for each cache 


You can also use JMX to monitor rebalance:
https://www.gridgain.com/docs/latest/administrators-guide/monitoring-metrics/metrics#monitoring-rebalancing

In general, it is advisable to clean old metadata when re-adding a baseline
node(especially if you made changes to you cache config), but be careful to
make sure that no data is lost inadvertently.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Embedded ignite and baseline upgrade questions

[djm132 <do...@gmail.com>]

!= in kotlin uses equals() under the hood so it works here as expected.

I can't use control.sh to manage topology (becase ignite is embedded) and
trying to implement it in code. So the actual initialization sequence is:

1) Start all fresh nodes, wait for all specified persistence nodes to be
online.
2) Auto-activation sets baseline to all nodes online to the moment.
3) If it first launch - currentBaselineTopology() will be null (or probably
different set) and I will set it to defined list of nodes using
setBaselineTopology().
4) Later if extending baseline required - we wait for all of them and call
setBaselineTopology to new list.
5) Later if shrinking baseline required - we set it using the same
setBaselineTopology() again (excluded node need to be offline for sure), it
also triggers data rebalancing as expected.
6) If we need to include new node to baseline, its definitely need to be
cleaned from old stored baseline metadata, right?

So it seems that managing topology by specifying a list in config file is
possible with this code? Or may be I am missing something ?

Is there any global event for data rebalancing ? Or it should be tracked for
every cache with localListen ?

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Embedded ignite and baseline upgrade questions

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

First of all, I really hope you're not using != to compare collections :)

Second, cluster will auto-activate when all baseline nodes have joined. For
the first time you have to manually activate the cluster.

The algorithm looks OK and nodes should auto-activate once they are all up.

With regards to your first question - you have managed nodes from different
baseline topologies, i.e., nodes from different clusters. Persistent node
from the "wrong" cluster will not join.

Regards,
-- 
Ilya Kasnacheev


сб, 28 дек. 2019 г. в 00:41, djm132 <do...@gmail.com>:

> Also when starting second node I am getting this:
>
> 2019/12/27 23:39:36.665 [disco-pool-#55] WARN BaselineTopology of joining
> node (dev-1) is not compatible with BaselineTopology in the cluster.
> Branching history of cluster BlT ([95475834]) doesn't contain branching
> point hash of joining node BlT (95475833). Consider cleaning persistent
> storage of the node and adding it to the cluster again.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Embedded ignite and baseline upgrade questions

[djm132 <do...@gmail.com>]

Also when starting second node I am getting this:

2019/12/27 23:39:36.665 [disco-pool-#55] WARN BaselineTopology of joining
node (dev-1) is not compatible with BaselineTopology in the cluster.
Branching history of cluster BlT ([95475834]) doesn't contain branching
point hash of joining node BlT (95475833). Consider cleaning persistent
storage of the node and adding it to the cluster again.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/