You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ode.apache.org by sudharma subasinghe <su...@cse.mrt.ac.lk> on 2015/05/14 11:59:24 UTC

Deploying process in the cluster

Hi,

In the cluster, every node shares the database and deployment directory. To
avoid race condition I am going to introduce master and slave
configuration. Here master node will deploy the process and write the
version number to the database. Until master node finishes its process all
the salve nodes will wait. After finishing the master node's task it will
send a custom message to slave nodes through the cluster. Then they can get
the version number from database and will read the deployed process to
deploy it in salve nodes. Please give me ideas on this.

Thanks,
Sudharma

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Sathwik,

I attached the patch to the JIRA ODE-563. All the changes are available in
the patch.

regards,
Sudharma

On 30 June 2015 at 18:43, Sathwik B P <sa...@gmail.com> wrote:

> Hi Sudharma,
>
> Can you contribute your code as patch and attach it to the JIRA ODE-563.
>
> You could create the patch file in this way.
> git diff master ODECluster > cluster_patch1.patch
>
> Make sure all your changes are available in the patch.
>
> regards,
> sathwik
>
> On Wed, Jun 17, 2015 at 12:51 PM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk
> > wrote:
>
> > Hi Sathwik,
> >
> > I modified the code as you explained. You can go through the code by
> > following link.
> > https://github.com/Subasinghe/ode/tree/ODECluster
> >
> > It would be helpful if you can provide a feedback on this.
> >
> > Thank you.
> >
> > On 10 June 2015 at 12:39, sudharma subasinghe <su...@cse.mrt.ac.lk>
> > wrote:
> >
> > > Hi Sathwik,
> > >
> > > I tried to implement your logic also. Following is the link for
> committed
> > > code upto now.
> > > https://github.com/Subasinghe/ode/tree/ODECluster
> > >
> > > I used info msgs to debugging as it can be observed easily in console.
> > > When releasing the lock acquired by poller or web service the state is
> > set
> > > to false as in my code. But sometimes it is set to true and at that
> time
> > > second server's value is also true.  Following is the related log for
> > each
> > > server.
> > >
> > > Server 1
> > > 02:12:02,957 INFO  [DeploymentPoller] Trying to access the lock for
> > > MagicSession
> > > 02:12:02,962 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value
> for
> > > MagicSession file after locking: true
> > > 02:12:03,838 INFO  [BpelServerImpl] Registered process {
> > > http://ode/bpel/unit-test}MagicSessionMain-3.
> > > 02:12:04,015 INFO  [BpelServerImpl] Registered process {
> > > http://ode/bpel/responder}MagicSessionResponder-3.
> > > 02:12:04,015 INFO  [DeploymentPoller] Deployment of artifact
> MagicSession
> > > successful: [{http://ode/bpel/unit-test}MagicSessionMain-3, {
> > > http://ode/bpel/responder}MagicSessionResponder-3]
> > > 02:12:04,015 INFO  [DeploymentPoller] Trying to release the lock for
> > > MagicSession
> > > 02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value
> for
> > > MagicSession file after unlocking: true
> > >
> > > Server 2
> > > 02:12:03,119 INFO  [DeploymentPoller] Trying to access the lock for
> > > MagicSession
> > > 02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value
> for
> > > MagicSession file after locking: true
> > > 02:12:04,017 INFO  [DeploymentPoller] Trying to release the lock for
> > > MagicSession
> > > 02:12:04,019 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value
> for
> > > MagicSession file after unlocking: false
> > >
> > > Is this possible? But there were not any conflicts while deploying. It
> > > worked perfectly.
> > >
> > > Thank you
> > >
> > >
> > > On 7 June 2015 at 11:10, sudharma subasinghe <su...@cse.mrt.ac.lk>
> > > wrote:
> > >
> > >> Hi Sathwik,
> > >>
> > >> I will concern your approach. Thank you for your effort.
> > >>
> > >> On 6 June 2015 at 18:43, Sathwik B P <sa...@gmail.com> wrote:
> > >>
> > >>> I have attached to the JIRA
> > >>> https://issues.apache.org/jira/browse/ODE-563
> > >>>
> > >>> On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <
> > >>> suba.11@cse.mrt.ac.lk>
> > >>> wrote:
> > >>>
> > >>> > Hi Sathwik,
> > >>> >
> > >>> > I can't find the attached document. Please kind be enough to resend
> > it.
> > >>> >
> > >>> > On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:
> > >>> >
> > >>> > > Refer this one as I have corrected the numbering of steps.
> > >>> > >
> > >>> > > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <
> sathwik.bp@gmail.com>
> > >>> > wrote:
> > >>> > >
> > >>> > >> Hi Sudharma/Tammo,
> > >>> > >>
> > >>> > >> Kindly review attached document on my proposed approach. Let me
> > >>> know if
> > >>> > >> you have any concerns or doubts on it.
> > >>> > >>
> > >>> > >> regards,
> > >>> > >> sathwik
> > >>> > >>
> > >>> > >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <
> sathwik.bp@gmail.com
> > >
> > >>> > wrote:
> > >>> > >>
> > >>> > >>> find response inline
> > >>> > >>>
> > >>> > >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
> > >>> > >>> suba.11@cse.mrt.ac.lk> wrote:
> > >>> > >>>
> > >>> > >>>> Hi,
> > >>> > >>>>
> > >>> > >>>> Sorry for the late reply. I was trying to achieve a proper
> > >>> solution.
> > >>> > >>>> Following is my approach.
> > >>> > >>>>
> > >>> > >>>> 1) I elected master node for deploying purpose. So when a
> master
> > >>> node
> > >>> > >>>> goes
> > >>> > >>>> down hazelcast will elect the next oldest node for master.
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>     Perfect
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>> 2) ODEServer can identify whether clustering is enabled or not
> > by
> > >>> > >>>> getting
> > >>> > >>>> the property value in ode-axis2.properties file. So I
> introduced
> > >>> new
> > >>> > >>>> property called "ode-axis2.hazelcast.clustering.enabled". If
> > >>> there is
> > >>> > no
> > >>> > >>>> clustering enabled server will work as it is. If clustering is
> > >>> > enabled,
> > >>> > >>>> cluster will be initialized.
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>     Perfect
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>> 3) In manual deployment its responsibility is taken by the
> > >>> > >>>> DeploymentPoller. So I give the deployment capability to
> master
> > >>> node,s
> > >>> > >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to
> > >>> true.So
> > >>> > >>>> others will not be able to go into check() method in
> > >>> DeploymentPoller.
> > >>> > >>>> So
> > >>> > >>>> deployment will be done by only the master node.
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>     Perfect
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>>
> > >>> > >>>> 4) In DeploymentWebService, I had to consider few cases.If the
> > >>> deploy
> > >>> > >>>> request goes to the master node, it will deploy the process
> > >>> through
> > >>> > web
> > >>> > >>>> service.Others pollers will not go into check() method as they
> > >>> are not
> > >>> > >>>> masters. So master can continue without any involvement of
> > others.
> > >>> > >>>>
> > >>> > >>>>
> > >>> > >>>     Perfect
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>> 5) If the deploy request goes to a slave node, it will do up
> to
> > >>> file
> > >>> > >>>> creation in the file system.Slave will be stopped at that
> point.
> > >>> As
> > >>> > only
> > >>> > >>>> master poller is checking, master can continue from created
> > files
> > >>> in
> > >>> > the
> > >>> > >>>> file system.
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>     DeploymentWebService provides synchronous operations. The
> > >>> status of
> > >>> > >>> the operation should be communicated to the calling client in
> the
> > >>> same
> > >>> > call.
> > >>> > >>>     DeploymentPoller is a backend thread that goes over each
> and
> > >>> every
> > >>> > >>> directory under the Deployment directory checking for any
> changes
> > >>> to
> > >>> > >>> deploy.xml in existing processes and deploy newly added
> > processes.
> > >>> This
> > >>> > >>> process is  sequential and time consuming. As the process
> > >>> directories
> > >>> > >>> grows, so does the time taken for execution of the thread.
> > >>> > >>>
> > >>> > >>>     Since the request is on a slave node and the processing is
> > >>> done on
> > >>> > >>> master node, how do you check for the completion of the
> > >>> > >>> deployment/undeployment of processes and respond back to the
> > client
> > >>> > since
> > >>> > >>> the web service call is a synchronous operation. As
> > >>> DeploymentPoller is
> > >>> > >>> taking a lot of time in processing, your request will time out
> > >>> right.
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>>
> > >>> > >>>> 6) But there was problem with _deploymentUnits in
> > >>> ProcessStoreImpl.
> > >>> > Each
> > >>> > >>>> _deploymentUnits stores only what its server has deployed. So
> > >>> think,
> > >>> > >>>> that a
> > >>> > >>>> master node goes down another master node appears.But its
> > >>> > >>>> __deploymentUnits
> > >>> > >>>> does not have dus which has deployed by the earlier master
> node.
> > >>> Hence
> > >>> > >>>> it
> > >>> > >>>> will not be able retire earlier version of the process which
> is
> > >>> > >>>> deployed by
> > >>> > >>>> previous master. So there will two process which are in
> "ACTIVE"
> > >>> state
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>> 7) To avoid this, I add the ODEServer as an Observer to check
> > >>> when a
> > >>> > new
> > >>> > >>>> master is electing, then load all the deployment units out of
> > the
> > >>> > >>>> store. So
> > >>> > >>>> new master node can have all the dus and can retire
> appropriate
> > >>> > version.
> > >>> > >>>> Usually loadAll() is called at the server start-up time. But
> > >>> there is
> > >>> > no
> > >>> > >>>> other way to solve this. I tried to use Hazelcast IMap to
> store
> > >>> all
> > >>> > dus
> > >>> > >>>> among all nodes. But it wasn't success as du is not
> serializable
> > >>> > >>>> object.
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>> 8) I figured out that we do not need send cluster message to
> > >>> others as
> > >>> > >>>> all
> > >>> > >>>> the dus' data are persisted to the shared DB. So each node can
> > >>> take
> > >>> > the
> > >>> > >>>> du
> > >>> > >>>> and retrieve necessary data using already implemented methods
> in
> > >>> > Process
> > >>> > >>>> Store.
> > >>> > >>>>
> > >>> > >>>> 9) But there is an another problem.The axis2 service
> > >>> corresponding to
> > >>> > a
> > >>> > >>>> deployed process does not appear on all nodes of the cluster.
> > >>> That is
> > >>> > >>>> because each server add du which is deployed by it to the
> > process
> > >>> > >>>> store.That is why I had use loadAll() when masters are
> changing.
> > >>> How
> > >>> > to
> > >>> > >>>> solve this?
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>     I do appriciate your efforts in understanding the
> > >>> implmentation and
> > >>> > >>> changes that need to be done. You are bang on it.
> > >>> > >>>     fireEvent(..) is the method that triggers process
> activation
> > >>> and
> > >>> > >>> necessary service creation.
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>     But with this given apporach from steps 6 to 8, ODE cannot
> > have
> > >>> > >>> atleast 2 Active servers for Load balancing. You are
> > concentrating
> > >>> on
> > >>> > only
> > >>> > >>> one active node that will do deployments and cater to process
> > >>> > invocations.
> > >>> > >>>     We should also think about scaling ODE to multiple servers
> to
> > >>> > handle
> > >>> > >>> load.
> > >>> > >>>
> > >>> > >>>     What do you think.
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>> Thank you,
> > >>> > >>>> Sudharma
> > >>> > >>>>
> > >>> > >>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com>
> > >>> wrote:
> > >>> > >>>>
> > >>> > >>>> > Sudharma,
> > >>> > >>>> >
> > >>> > >>>> > Any updates?
> > >>> > >>>> >
> > >>> > >>>> > regards,
> > >>> > >>>> > sathwik
> > >>> > >>>> >
> > >>> > >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <
> > >>> sathwik.bp@gmail.com>
> > >>> > >>>> wrote:
> > >>> > >>>> >
> > >>> > >>>> > > Sudharma,
> > >>> > >>>> > >
> > >>> > >>>> > > Can you elaborate on your option 1).
> > >>> > >>>> > >
> > >>> > >>>> > > Response to your option 2).
> > >>> > >>>> > >
> > >>> > >>>> > >     Process Store is the component that handles process
> > >>> metadata,
> > >>> > >>>> > > compilation and deployment in ODE. Integration layers in
> ODE
> > >>> > >>>> (Axis2, JBI)
> > >>> > >>>> > > use the process store.
> > >>> > >>>> > >     Future implementations of IL for ODE will also use the
> > >>> process
> > >>> > >>>> store.
> > >>> > >>>> > > We should not be thinking of moving the process store
> > >>> > functionality
> > >>> > >>>> to
> > >>> > >>>> > the
> > >>> > >>>> > > integration layers.
> > >>> > >>>> > >
> > >>> > >>>> > >
> > >>> > >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> > >>> > >>>> > > suba.11@cse.mrt.ac.lk> wrote:
> > >>> > >>>> > >
> > >>> > >>>> > >> Hi,
> > >>> > >>>> > >>
> > >>> > >>>> > >> I understood the problem within dynamic master/slave
> > >>> > >>>> configuration. In
> > >>> > >>>> > my
> > >>> > >>>> > >> approach, when a deployment request is routed to a slave
> > node
> > >>> > >>>> there will
> > >>> > >>>> > >> not be a deployment. I suggest two options to avoid it.
> > >>> > >>>> > >> 1) Have static master/slave configuration only for deploy
> > >>> process
> > >>> > >>>> > >>
> > >>> > >>>> > > 2) Modify the deployment web service to complie and verify
> > the
> > >>> > >>>> process
> > >>> > >>>> > and
> > >>> > >>>> > >> then copy it to the deploy folder irrespective of whether
> > >>> its a
> > >>> > >>>> master
> > >>> > >>>> > or
> > >>> > >>>> > >> slave, then deployment poller should take care of the
> > >>> deployment
> > >>> > >>>> > >>
> > >>> > >>>> > >>
> > >>> > >>>> > >
> > >>> > >>>> > >>
> > >>> > >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <
> sathwik.bp@gmail.com
> > >
> > >>> > wrote:
> > >>> > >>>> > >>
> > >>> > >>>> > >> > Sudharma,
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > We definitely need a master/slave in the hazelcast
> > cluster.
> > >>> > This
> > >>> > >>>> is
> > >>> > >>>> > >> > probably needed for the job migration in the Scheduler
> to
> > >>> > >>>> migrate the
> > >>> > >>>> > >> jobs
> > >>> > >>>> > >> > associated with a down node. Let hold on this topic for
> > >>> future
> > >>> > >>>> > >> discussion.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > Going by the explanation where the master/slave nodes
> > have
> > >>> > >>>> certain
> > >>> > >>>> > >> > predefined tasks to perform is perfectly fine.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > I have this scenario,
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > I am using HAProxy as my load balancer and configured 3
> > >>> nodes
> > >>> > in
> > >>> > >>>> the
> > >>> > >>>> > >> > cluster.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > Node1 - Active
> > >>> > >>>> > >> > Node2 - Active
> > >>> > >>>> > >> > Node3 - Backup
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > Load balancing algorithm: RoundRobin
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > A Backup node (Node3) is one which the load balancer
> will
> > >>> not
> > >>> > >>>> route
> > >>> > >>>> > >> > requests to, until one of the Active node i.e either
> > Node1
> > >>> or
> > >>> > >>>> Node2
> > >>> > >>>> > has
> > >>> > >>>> > >> > gone down.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > All these 3 nodes are also part of the hazelcast
> cluster
> > as
> > >>> > well.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > In the hazelcast cluster, assume Node1 is elected as
> the
> > >>> > >>>> leader/master
> > >>> > >>>> > >> and
> > >>> > >>>> > >> > Node2,Node3 as slaves.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > I initiate the deploy operation on the
> > DeploymentWebService
> > >>> > >>>> which the
> > >>> > >>>> > >> load
> > >>> > >>>> > >> > balancer routes it to one of the Active nodes in the
> > >>> cluster,
> > >>> > >>>> lets say
> > >>> > >>>> > >> it's
> > >>> > >>>> > >> > the Node1. Since Node1 is also the master in the
> > hazelcast
> > >>> > >>>> cluster,
> > >>> > >>>> > >> > deployment is a success.
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > I initiate another deploy operation on the
> > >>> DeploymentWebService
> > >>> > >>>> which
> > >>> > >>>> > >> the
> > >>> > >>>> > >> > load balancer routes it to the next active node which
> is
> > >>> Node2.
> > >>> > >>>> Since
> > >>> > >>>> > >> Node2
> > >>> > >>>> > >> > is a slave in the Hazelcast cluster, What happens to
> the
> > >>> > >>>> deployment?
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > regards,
> > >>> > >>>> > >> > sathwik
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> > >>> > >>>> > >> > suba.11@cse.mrt.ac.lk
> > >>> > >>>> > >> > > wrote:
> > >>> > >>>> > >> >
> > >>> > >>>> > >> > > Hi,
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > > I will explain my approach as much as possible. The
> > >>> oldest
> > >>> > >>>> node in
> > >>> > >>>> > the
> > >>> > >>>> > >> > > hazelcast cluster is elected as the master node. In
> the
> > >>> > >>>> failure of
> > >>> > >>>> > the
> > >>> > >>>> > >> > > master node, next oldest node will be elected as the
> > >>> master
> > >>> > >>>> node.
> > >>> > >>>> > This
> > >>> > >>>> > >> > > master-slave configuration is just for deployment.
> When
> > >>> the
> > >>> > >>>> > hazelcast
> > >>> > >>>> > >> > > cluster elected the master node, that node becomes a
> > >>> master
> > >>> > >>>> node for
> > >>> > >>>> > >> > > deploying process. So it will do the deploying
> > >>> artifacts. If
> > >>> > >>>> you
> > >>> > >>>> > want
> > >>> > >>>> > >> to
> > >>> > >>>> > >> > > get the idea of electing master node please refer the
> > >>> code
> > >>> > >>>> which I
> > >>> > >>>> > >> have
> > >>> > >>>> > >> > > located in the github. (
> > >>> > >>>> > >> > >
> https://github.com/Subasinghe/ode/tree/ode_clustering)
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > > I identified separated actions which should be
> followed
> > >>> by
> > >>> > the
> > >>> > >>>> > master
> > >>> > >>>> > >> and
> > >>> > >>>> > >> > > salve nodes.
> > >>> > >>>> > >> > > Actions which are followed by master node only
> > >>> > >>>> > >> > > 1) create deployment unit
> > >>> > >>>> > >> > > 2) set the version nu to deployment unit
> > >>> > >>>> > >> > > 3) compile deployment unit
> > >>> > >>>> > >> > > 4) scan deployment unit
> > >>> > >>>> > >> > > 5) retire previous versions
> > >>> > >>>> > >> > > Master node and slave nodes should create _processes
> > >>> which
> > >>> > >>>> stores
> > >>> > >>>> > >> > > ProcessConfImpl
> > >>> > >>>> > >> > > Only master node will write the version nu to
> database,
> > >>> > create
> > >>> > >>>> > >> .deployed
> > >>> > >>>> > >> > > file
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > > So there are some actions which should be followed
> only
> > >>> by
> > >>> > >>>> master
> > >>> > >>>> > node
> > >>> > >>>> > >> > > while other actions should be followed by all the
> > >>> nodes.The
> > >>> > >>>> idea of
> > >>> > >>>> > >> > having
> > >>> > >>>> > >> > > a master node is deploying artifacts and avoid others
> > >>> from
> > >>> > >>>> writing
> > >>> > >>>> > the
> > >>> > >>>> > >> > > version nu to database.
> > >>> > >>>> > >> > > Whether a node is active or passive, all nodes should
> > do
> > >>> the
> > >>> > >>>> > >> > > deployment.Master
> > >>> > >>>> > >> > > and slaves will follow necessary actions as in above.
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <
> > >>> sathwik.bp@gmail.com>
> > >>> > >>>> wrote:
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> > > > Nandika,
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > I very well understand what you have put across,
> but
> > >>> it's
> > >>> > >>>> > secondary
> > >>> > >>>> > >> to
> > >>> > >>>> > >> > me
> > >>> > >>>> > >> > > > now.
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > Sudharma,
> > >>> > >>>> > >> > > > My primary concern is to understand at a high level
> > the
> > >>> > >>>> deployment
> > >>> > >>>> > >> > > > architecture and how would master-slave
> configuration
> > >>> fit
> > >>> > >>>> in. Are
> > >>> > >>>> > >> there
> > >>> > >>>> > >> > > any
> > >>> > >>>> > >> > > > restrictions imposed by the in-progress design?
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > Firstly, how would ODE process deployment work
> under
> > >>> these
> > >>> > >>>> cluster
> > >>> > >>>> > >> > > > configurations?
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > Sample Cluster configurations: A load balancer is
> > >>> > >>>> frontending the
> > >>> > >>>> > >> > > servers.
> > >>> > >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> > >>> > >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> > >>> > >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes
> either
> > >>> in
> > >>> > >>>> Active or
> > >>> > >>>> > >> > > Passive.
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > regards,
> > >>> > >>>> > >> > > > sathwik
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika
> Jayawardana
> > <
> > >>> > >>>> > >> > jayawark@gmail.com
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > wrote:
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > > > > Hi Sathwik,
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > > According to my understanding, in the clustering
> > >>> > scenario,
> > >>> > >>>> the
> > >>> > >>>> > >> master
> > >>> > >>>> > >> > > > node
> > >>> > >>>> > >> > > > > should perform all the deployment actions and the
> > >>> slave
> > >>> > >>>> nodes
> > >>> > >>>> > also
> > >>> > >>>> > >> > need
> > >>> > >>>> > >> > > > to
> > >>> > >>>> > >> > > > > perform some deployment actions. For example, the
> > >>> slave
> > >>> > >>>> nodes
> > >>> > >>>> > also
> > >>> > >>>> > >> > > should
> > >>> > >>>> > >> > > > > handle the process ACTIVATED event so that the
> > >>> process
> > >>> > >>>> > >> configuration
> > >>> > >>>> > >> > is
> > >>> > >>>> > >> > > > > added to the engine and necessary web services
> are
> > >>> > created
> > >>> > >>>> so
> > >>> > >>>> > that
> > >>> > >>>> > >> > when
> > >>> > >>>> > >> > > > the
> > >>> > >>>> > >> > > > > load balancer send requests to any node in the
> > >>> cluster,
> > >>> > it
> > >>> > >>>> is
> > >>> > >>>> > >> ready
> > >>> > >>>> > >> > to
> > >>> > >>>> > >> > > > > accept those requests.
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > > Regards
> > >>> > >>>> > >> > > > > Nandika
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> > >>> > >>>> > >> sathwik.bp@gmail.com>
> > >>> > >>>> > >> > > > > wrote:
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > > > > Sudharma,
> > >>> > >>>> > >> > > > > >
> > >>> > >>>> > >> > > > > > Where are you going to configure the
> > >>> master-slaves, is
> > >>> > >>>> it in
> > >>> > >>>> > the
> > >>> > >>>> > >> > web
> > >>> > >>>> > >> > > > > > application level or at the load balancer?
> > >>> > >>>> > >> > > > > >
> > >>> > >>>> > >> > > > > > regards,
> > >>> > >>>> > >> > > > > > sathwik
> > >>> > >>>> > >> > > > > >
> > >>> > >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma
> > >>> subasinghe <
> > >>> > >>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
> > >>> > >>>> > >> > > > > > wrote:
> > >>> > >>>> > >> > > > > >
> > >>> > >>>> > >> > > > > > > Hi Tammo,
> > >>> > >>>> > >> > > > > > >
> > >>> > >>>> > >> > > > > > > Can you suggest the best method from these to
> > >>> > >>>> implement? As
> > >>> > >>>> > >> > first I
> > >>> > >>>> > >> > > > > > > suggested the master-slaves scenario I think
> it
> > >>> is
> > >>> > >>>> easy to
> > >>> > >>>> > >> > > implement
> > >>> > >>>> > >> > > > > than
> > >>> > >>>> > >> > > > > > > distributed lock scenario. However if you can
> > >>> suggest
> > >>> > >>>> one
> > >>> > >>>> > from
> > >>> > >>>> > >> > > these
> > >>> > >>>> > >> > > > > two,
> > >>> > >>>> > >> > > > > > > then I can think about it.
> > >>> > >>>> > >> > > > > > >
> > >>> > >>>> > >> > > > > > > Thank you
> > >>> > >>>> > >> > > > > > >
> > >>> > >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
> > >>> > >>>> sathwik.bp@gmail.com>
> > >>> > >>>> > >> > wrote:
> > >>> > >>>> > >> > > > > > >
> > >>> > >>>> > >> > > > > > > > With respect to the hotdeployment,
> > >>> > >>>> > >> > > > > > > >
> > >>> > >>>> > >> > > > > > > > We can drop the deployment archive onto the
> > >>> > >>>> deployment
> > >>> > >>>> > >> folder.
> > >>> > >>>> > >> > > > Since
> > >>> > >>>> > >> > > > > > the
> > >>> > >>>> > >> > > > > > > > DeploymentPoller are acquiring the
> > distributed
> > >>> lock
> > >>> > >>>> for
> > >>> > >>>> > the
> > >>> > >>>> > >> > > > > > > DeploymentUnit,
> > >>> > >>>> > >> > > > > > > > only one of the nodes will get the lock and
> > >>> > initiate
> > >>> > >>>> the
> > >>> > >>>> > >> > > > deployment.
> > >>> > >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail
> in
> > >>> > >>>> acquiring
> > >>> > >>>> > the
> > >>> > >>>> > >> > lock
> > >>> > >>>> > >> > > > and
> > >>> > >>>> > >> > > > > > > hence
> > >>> > >>>> > >> > > > > > > > will silently ignore it.
> > >>> > >>>> > >> > > > > > > >
> > >>> > >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B
> > P <
> > >>> > >>>> > >> > > > sathwik.bp@gmail.com>
> > >>> > >>>> > >> > > > > > > > wrote:
> > >>> > >>>> > >> > > > > > > >
> > >>> > >>>> > >> > > > > > > > > Hi Tammo,
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > The distributed lock acquisition on the
> > >>> > >>>> DeploymentUnit
> > >>> > >>>> > >> should
> > >>> > >>>> > >> > > be
> > >>> > >>>> > >> > > > > > added
> > >>> > >>>> > >> > > > > > > to
> > >>> > >>>> > >> > > > > > > > > both DeploymentWebService and
> > >>> DeploymentPoller.
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > When a deployment operation is initiated
> > >>> through
> > >>> > >>>> the
> > >>> > >>>> > >> > > > > > > > DeploymentWebService,
> > >>> > >>>> > >> > > > > > > > > The load balancer routes it to any of the
> > >>> > available
> > >>> > >>>> > nodes.
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > On the routed node, the
> > DeploymentWebService
> > >>> > >>>> acquires
> > >>> > >>>> > the
> > >>> > >>>> > >> > > > > Distributed
> > >>> > >>>> > >> > > > > > > > > lock. On the remaining nodes the
> > >>> DeploymentPoller
> > >>> > >>>> will
> > >>> > >>>> > >> try to
> > >>> > >>>> > >> > > > > acquire
> > >>> > >>>> > >> > > > > > > the
> > >>> > >>>> > >> > > > > > > > > distributed lock and will not get it and
> > >>> hence
> > >>> > will
> > >>> > >>>> > >> silently
> > >>> > >>>> > >> > > > ignore
> > >>> > >>>> > >> > > > > > it.
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > Once the routed node completes the
> > >>> deployment, it
> > >>> > >>>> will
> > >>> > >>>> > >> > release
> > >>> > >>>> > >> > > > the
> > >>> > >>>> > >> > > > > > > lock.
> > >>> > >>>> > >> > > > > > > > > This way we don't have to stall the
> > >>> > >>>> DeploymentPoller in
> > >>> > >>>> > >> other
> > >>> > >>>> > >> > > > > nodes.
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > Does it answer the concerns?
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > Now, if we give the responsibility of
> > >>> identifying
> > >>> > >>>> the
> > >>> > >>>> > >> master
> > >>> > >>>> > >> > > node
> > >>> > >>>> > >> > > > > to
> > >>> > >>>> > >> > > > > > > the
> > >>> > >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the
> > >>> load
> > >>> > >>>> balancer
> > >>> > >>>> > to
> > >>> > >>>> > >> > > change
> > >>> > >>>> > >> > > > > > it's
> > >>> > >>>> > >> > > > > > > > > configuration about the master node?
> > >>> > >>>> > >> > > > > > > > > Assuming there are 3 nodes in the
> cluster,
> > >>> > >>>> > >> > > > > > > > > node1 -master
> > >>> > >>>> > >> > > > > > > > > node2 - slave
> > >>> > >>>> > >> > > > > > > > > node3 - slave
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > Node1 goes down, the LB will promote
> Node2
> > as
> > >>> > >>>> master
> > >>> > >>>> > node,
> > >>> > >>>> > >> > but
> > >>> > >>>> > >> > > > > > > hazelcast
> > >>> > >>>> > >> > > > > > > > > might promote Node3 as master node. They
> > are
> > >>> out
> > >>> > of
> > >>> > >>>> > sync.
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > Is this argument valid?
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > regards,
> > >>> > >>>> > >> > > > > > > > > sathwik
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo
> van
> > >>> > Lessen <
> > >>> > >>>> > >> > > > > > > tvanlessen@gmail.com>
> > >>> > >>>> > >> > > > > > > > > wrote:
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > >> Hi Sudharma,
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >> what do you expect from the "other nodes
> > >>> > >>>> deployment"?
> > >>> > >>>> > >> > > > Compilation
> > >>> > >>>> > >> > > > > is
> > >>> > >>>> > >> > > > > > > not
> > >>> > >>>> > >> > > > > > > > >> needed since the CBP file is written to
> > the
> > >>> > >>>> (shared)
> > >>> > >>>> > FS.
> > >>> > >>>> > >> > > > > > Registration
> > >>> > >>>> > >> > > > > > > is
> > >>> > >>>> > >> > > > > > > > >> also not needed, since it is done via
> the
> > >>> shared
> > >>> > >>>> > >> database.
> > >>> > >>>> > >> > So
> > >>> > >>>> > >> > > > the
> > >>> > >>>> > >> > > > > > only
> > >>> > >>>> > >> > > > > > > > >> thing that might be needed is to tell
> the
> > >>> engine
> > >>> > >>>> that
> > >>> > >>>> > >> there
> > >>> > >>>> > >> > > is a
> > >>> > >>>> > >> > > > > new
> > >>> > >>>> > >> > > > > > > > >> deployment. I'd need to check that. If
> > this
> > >>> is
> > >>> > >>>> needed,
> > >>> > >>>> > I
> > >>> > >>>> > >> > > revert
> > >>> > >>>> > >> > > > my
> > >>> > >>>> > >> > > > > > > last
> > >>> > >>>> > >> > > > > > > > >> statement, then it is perhaps better to
> > just
> > >>> > send
> > >>> > >>>> an
> > >>> > >>>> > >> event
> > >>> > >>>> > >> > > over
> > >>> > >>>> > >> > > > > > > > Hazelcast
> > >>> > >>>> > >> > > > > > > > >> to all nodes that the deployment has
> > >>> changed.
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >> Best,
> > >>> > >>>> > >> > > > > > > > >>   Tammo
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM,
> sudharma
> > >>> > >>>> subasinghe <
> > >>> > >>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
> > >>> > >>>> > >> > > > > > > > >> > wrote:
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >> > Hi Tammo,
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >> > The master node writes meta data. But
> > >>> runtime
> > >>> > >>>> > >> information
> > >>> > >>>> > >> > > must
> > >>> > >>>> > >> > > > > be
> > >>> > >>>> > >> > > > > > > > >> available
> > >>> > >>>> > >> > > > > > > > >> > in all nodes.Since the folder is
> shared,
> > >>> all
> > >>> > >>>> nodes
> > >>> > >>>> > will
> > >>> > >>>> > >> > see
> > >>> > >>>> > >> > > > the
> > >>> > >>>> > >> > > > > > > > >> > availability of a new process. My idea
> > is
> > >>> for
> > >>> > >>>> master
> > >>> > >>>> > >> node
> > >>> > >>>> > >> > to
> > >>> > >>>> > >> > > > > write
> > >>> > >>>> > >> > > > > > > the
> > >>> > >>>> > >> > > > > > > > >> meta
> > >>> > >>>> > >> > > > > > > > >> > data and other nodes to just read the
> > meta
> > >>> > data
> > >>> > >>>> and
> > >>> > >>>> > >> load
> > >>> > >>>> > >> > > > > > process.So
> > >>> > >>>> > >> > > > > > > we
> > >>> > >>>> > >> > > > > > > > >> need
> > >>> > >>>> > >> > > > > > > > >> > a small delay between master node
> > >>> deployment
> > >>> > and
> > >>> > >>>> > other
> > >>> > >>>> > >> > nodes
> > >>> > >>>> > >> > > > > > > > deployment.
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >> > Is there anyway to set the delay
> between
> > >>> > master
> > >>> > >>>> node
> > >>> > >>>> > >> and
> > >>> > >>>> > >> > > > slaves
> > >>> > >>>> > >> > > > > > > until
> > >>> > >>>> > >> > > > > > > > >> > master node finish the deployment?
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >> > Thank you
> > >>> > >>>> > >> > > > > > > > >> > Sudharma
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van
> > Lessen
> > >>> <
> > >>> > >>>> > >> > > > tvanlessen@gmail.com
> > >>> > >>>> > >> > > > > >
> > >>> > >>>> > >> > > > > > > > wrote:
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >> > > Hi Sathwik,
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM,
> > >>> Sathwik B
> > >>> > P <
> > >>> > >>>> > >> > > > > > > sathwik.bp@gmail.com>
> > >>> > >>>> > >> > > > > > > > >> > wrote:
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
> > >>> > >>>> > >> > > > > > > > >> > > >
> > >>> > >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which
> is
> > >>> the
> > >>> > >>>> master
> > >>> > >>>> > >> node
> > >>> > >>>> > >> > in
> > >>> > >>>> > >> > > > the
> > >>> > >>>> > >> > > > > > > > cluster?
> > >>> > >>>> > >> > > > > > > > >> > > >
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > I think the easiest approach is to
> > >>> always
> > >>> > >>>> elect the
> > >>> > >>>> > >> > oldest
> > >>> > >>>> > >> > > > > node
> > >>> > >>>> > >> > > > > > in
> > >>> > >>>> > >> > > > > > > > the
> > >>> > >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK
> > >>> Hazelcast
> > >>> > can
> > >>> > >>>> > easily
> > >>> > >>>> > >> > asked
> > >>> > >>>> > >> > > > for
> > >>> > >>>> > >> > > > > > > this
> > >>> > >>>> > >> > > > > > > > >> > > information.
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the
> > >>> Deployment
> > >>> > >>>> Pollers
> > >>> > >>>> > in
> > >>> > >>>> > >> > the
> > >>> > >>>> > >> > > > > slave
> > >>> > >>>> > >> > > > > > > > nodes?
> > >>> > >>>> > >> > > > > > > > >> > > >
> > >>> > >>>> > >> > > > > > > > >> > > >
> > >>> > >>>> > >> > > > > > > > >> > > Absolutely.
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > Suggestion:
> > >>> > >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
> > >>> > >>>> Master-SLaves.
> > >>> > >>>> > Why
> > >>> > >>>> > >> > not
> > >>> > >>>> > >> > > > give
> > >>> > >>>> > >> > > > > > > every
> > >>> > >>>> > >> > > > > > > > >> node
> > >>> > >>>> > >> > > > > > > > >> > > in
> > >>> > >>>> > >> > > > > > > > >> > > > the cluster the same status
> > >>> > (Active-Active).
> > >>> > >>>> > >> > > > > > > > >> > > >
> > >>> > >>>> > >> > > > > > > > >> > > > When a new deployment is made, the
> > >>> load
> > >>> > >>>> balancer
> > >>> > >>>> > >> can
> > >>> > >>>> > >> > > push
> > >>> > >>>> > >> > > > it
> > >>> > >>>> > >> > > > > > to
> > >>> > >>>> > >> > > > > > > > any
> > >>> > >>>> > >> > > > > > > > >> of
> > >>> > >>>> > >> > > > > > > > >> > > the
> > >>> > >>>> > >> > > > > > > > >> > > > available nodes. That node will
> > >>> probably
> > >>> > >>>> acquire
> > >>> > >>>> > a
> > >>> > >>>> > >> > > > > distributed
> > >>> > >>>> > >> > > > > > > > lock
> > >>> > >>>> > >> > > > > > > > >> on
> > >>> > >>>> > >> > > > > > > > >> > > the
> > >>> > >>>> > >> > > > > > > > >> > > > deployment unit and acts as master
> > for
> > >>> > that
> > >>> > >>>> > >> > deployment.
> > >>> > >>>> > >> > > > This
> > >>> > >>>> > >> > > > > > > > ensures
> > >>> > >>>> > >> > > > > > > > >> > > > optimum usage of the cluster
> nodes.
> > >>> > >>>> Probably no
> > >>> > >>>> > >> static
> > >>> > >>>> > >> > > > > > > > >> configuration of
> > >>> > >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer
> > nor
> > >>> in
> > >>> > the
> > >>> > >>>> > >> > hazelcast.
> > >>> > >>>> > >> > > > > > > > >> > > >
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > But this would not allow to have the
> > >>> > >>>> hotdeployment
> > >>> > >>>> > >> via
> > >>> > >>>> > >> > > > > > filesystem
> > >>> > >>>> > >> > > > > > > > >> still
> > >>> > >>>> > >> > > > > > > > >> > > enabled, right?
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > Best,
> > >>> > >>>> > >> > > > > > > > >> > >   Tammo
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> > > --
> > >>> > >>>> > >> > > > > > > > >> > > Tammo van Lessen -
> > http://www.taval.de
> > >>> > >>>> > >> > > > > > > > >> > >
> > >>> > >>>> > >> > > > > > > > >> >
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >> --
> > >>> > >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> > >>> > >>>> > >> > > > > > > > >>
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > > >
> > >>> > >>>> > >> > > > > > > >
> > >>> > >>>> > >> > > > > > >
> > >>> > >>>> > >> > > > > >
> > >>> > >>>> > >> > > > >
> > >>> > >>>> > >> > > >
> > >>> > >>>> > >> > >
> > >>> > >>>> > >> >
> > >>> > >>>> > >>
> > >>> > >>>> > >
> > >>> > >>>> > >
> > >>> > >>>> >
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>>
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Hi Sudharma,

Can you contribute your code as patch and attach it to the JIRA ODE-563.

You could create the patch file in this way.
git diff master ODECluster > cluster_patch1.patch

Make sure all your changes are available in the patch.

regards,
sathwik

On Wed, Jun 17, 2015 at 12:51 PM, sudharma subasinghe <suba.11@cse.mrt.ac.lk
> wrote:

> Hi Sathwik,
>
> I modified the code as you explained. You can go through the code by
> following link.
> https://github.com/Subasinghe/ode/tree/ODECluster
>
> It would be helpful if you can provide a feedback on this.
>
> Thank you.
>
> On 10 June 2015 at 12:39, sudharma subasinghe <su...@cse.mrt.ac.lk>
> wrote:
>
> > Hi Sathwik,
> >
> > I tried to implement your logic also. Following is the link for committed
> > code upto now.
> > https://github.com/Subasinghe/ode/tree/ODECluster
> >
> > I used info msgs to debugging as it can be observed easily in console.
> > When releasing the lock acquired by poller or web service the state is
> set
> > to false as in my code. But sometimes it is set to true and at that time
> > second server's value is also true.  Following is the related log for
> each
> > server.
> >
> > Server 1
> > 02:12:02,957 INFO  [DeploymentPoller] Trying to access the lock for
> > MagicSession
> > 02:12:02,962 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value for
> > MagicSession file after locking: true
> > 02:12:03,838 INFO  [BpelServerImpl] Registered process {
> > http://ode/bpel/unit-test}MagicSessionMain-3.
> > 02:12:04,015 INFO  [BpelServerImpl] Registered process {
> > http://ode/bpel/responder}MagicSessionResponder-3.
> > 02:12:04,015 INFO  [DeploymentPoller] Deployment of artifact MagicSession
> > successful: [{http://ode/bpel/unit-test}MagicSessionMain-3, {
> > http://ode/bpel/responder}MagicSessionResponder-3]
> > 02:12:04,015 INFO  [DeploymentPoller] Trying to release the lock for
> > MagicSession
> > 02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value for
> > MagicSession file after unlocking: true
> >
> > Server 2
> > 02:12:03,119 INFO  [DeploymentPoller] Trying to access the lock for
> > MagicSession
> > 02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value for
> > MagicSession file after locking: true
> > 02:12:04,017 INFO  [DeploymentPoller] Trying to release the lock for
> > MagicSession
> > 02:12:04,019 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value for
> > MagicSession file after unlocking: false
> >
> > Is this possible? But there were not any conflicts while deploying. It
> > worked perfectly.
> >
> > Thank you
> >
> >
> > On 7 June 2015 at 11:10, sudharma subasinghe <su...@cse.mrt.ac.lk>
> > wrote:
> >
> >> Hi Sathwik,
> >>
> >> I will concern your approach. Thank you for your effort.
> >>
> >> On 6 June 2015 at 18:43, Sathwik B P <sa...@gmail.com> wrote:
> >>
> >>> I have attached to the JIRA
> >>> https://issues.apache.org/jira/browse/ODE-563
> >>>
> >>> On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <
> >>> suba.11@cse.mrt.ac.lk>
> >>> wrote:
> >>>
> >>> > Hi Sathwik,
> >>> >
> >>> > I can't find the attached document. Please kind be enough to resend
> it.
> >>> >
> >>> > On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:
> >>> >
> >>> > > Refer this one as I have corrected the numbering of steps.
> >>> > >
> >>> > > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > >> Hi Sudharma/Tammo,
> >>> > >>
> >>> > >> Kindly review attached document on my proposed approach. Let me
> >>> know if
> >>> > >> you have any concerns or doubts on it.
> >>> > >>
> >>> > >> regards,
> >>> > >> sathwik
> >>> > >>
> >>> > >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sathwik.bp@gmail.com
> >
> >>> > wrote:
> >>> > >>
> >>> > >>> find response inline
> >>> > >>>
> >>> > >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
> >>> > >>> suba.11@cse.mrt.ac.lk> wrote:
> >>> > >>>
> >>> > >>>> Hi,
> >>> > >>>>
> >>> > >>>> Sorry for the late reply. I was trying to achieve a proper
> >>> solution.
> >>> > >>>> Following is my approach.
> >>> > >>>>
> >>> > >>>> 1) I elected master node for deploying purpose. So when a master
> >>> node
> >>> > >>>> goes
> >>> > >>>> down hazelcast will elect the next oldest node for master.
> >>> > >>>>
> >>> > >>>
> >>> > >>>     Perfect
> >>> > >>>
> >>> > >>>
> >>> > >>>> 2) ODEServer can identify whether clustering is enabled or not
> by
> >>> > >>>> getting
> >>> > >>>> the property value in ode-axis2.properties file. So I introduced
> >>> new
> >>> > >>>> property called "ode-axis2.hazelcast.clustering.enabled". If
> >>> there is
> >>> > no
> >>> > >>>> clustering enabled server will work as it is. If clustering is
> >>> > enabled,
> >>> > >>>> cluster will be initialized.
> >>> > >>>>
> >>> > >>>
> >>> > >>>     Perfect
> >>> > >>>
> >>> > >>>
> >>> > >>>> 3) In manual deployment its responsibility is taken by the
> >>> > >>>> DeploymentPoller. So I give the deployment capability to master
> >>> node,s
> >>> > >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to
> >>> true.So
> >>> > >>>> others will not be able to go into check() method in
> >>> DeploymentPoller.
> >>> > >>>> So
> >>> > >>>> deployment will be done by only the master node.
> >>> > >>>>
> >>> > >>>
> >>> > >>>     Perfect
> >>> > >>>
> >>> > >>>
> >>> > >>>>
> >>> > >>>> 4) In DeploymentWebService, I had to consider few cases.If the
> >>> deploy
> >>> > >>>> request goes to the master node, it will deploy the process
> >>> through
> >>> > web
> >>> > >>>> service.Others pollers will not go into check() method as they
> >>> are not
> >>> > >>>> masters. So master can continue without any involvement of
> others.
> >>> > >>>>
> >>> > >>>>
> >>> > >>>     Perfect
> >>> > >>>
> >>> > >>>
> >>> > >>>> 5) If the deploy request goes to a slave node, it will do up to
> >>> file
> >>> > >>>> creation in the file system.Slave will be stopped at that point.
> >>> As
> >>> > only
> >>> > >>>> master poller is checking, master can continue from created
> files
> >>> in
> >>> > the
> >>> > >>>> file system.
> >>> > >>>>
> >>> > >>>
> >>> > >>>     DeploymentWebService provides synchronous operations. The
> >>> status of
> >>> > >>> the operation should be communicated to the calling client in the
> >>> same
> >>> > call.
> >>> > >>>     DeploymentPoller is a backend thread that goes over each and
> >>> every
> >>> > >>> directory under the Deployment directory checking for any changes
> >>> to
> >>> > >>> deploy.xml in existing processes and deploy newly added
> processes.
> >>> This
> >>> > >>> process is  sequential and time consuming. As the process
> >>> directories
> >>> > >>> grows, so does the time taken for execution of the thread.
> >>> > >>>
> >>> > >>>     Since the request is on a slave node and the processing is
> >>> done on
> >>> > >>> master node, how do you check for the completion of the
> >>> > >>> deployment/undeployment of processes and respond back to the
> client
> >>> > since
> >>> > >>> the web service call is a synchronous operation. As
> >>> DeploymentPoller is
> >>> > >>> taking a lot of time in processing, your request will time out
> >>> right.
> >>> > >>>
> >>> > >>>
> >>> > >>>>
> >>> > >>>> 6) But there was problem with _deploymentUnits in
> >>> ProcessStoreImpl.
> >>> > Each
> >>> > >>>> _deploymentUnits stores only what its server has deployed. So
> >>> think,
> >>> > >>>> that a
> >>> > >>>> master node goes down another master node appears.But its
> >>> > >>>> __deploymentUnits
> >>> > >>>> does not have dus which has deployed by the earlier master node.
> >>> Hence
> >>> > >>>> it
> >>> > >>>> will not be able retire earlier version of the process which is
> >>> > >>>> deployed by
> >>> > >>>> previous master. So there will two process which are in "ACTIVE"
> >>> state
> >>> > >>>
> >>> > >>>
> >>> > >>>> 7) To avoid this, I add the ODEServer as an Observer to check
> >>> when a
> >>> > new
> >>> > >>>> master is electing, then load all the deployment units out of
> the
> >>> > >>>> store. So
> >>> > >>>> new master node can have all the dus and can retire appropriate
> >>> > version.
> >>> > >>>> Usually loadAll() is called at the server start-up time. But
> >>> there is
> >>> > no
> >>> > >>>> other way to solve this. I tried to use Hazelcast IMap to store
> >>> all
> >>> > dus
> >>> > >>>> among all nodes. But it wasn't success as du is not serializable
> >>> > >>>> object.
> >>> > >>>>
> >>> > >>>
> >>> > >>>> 8) I figured out that we do not need send cluster message to
> >>> others as
> >>> > >>>> all
> >>> > >>>> the dus' data are persisted to the shared DB. So each node can
> >>> take
> >>> > the
> >>> > >>>> du
> >>> > >>>> and retrieve necessary data using already implemented methods in
> >>> > Process
> >>> > >>>> Store.
> >>> > >>>>
> >>> > >>>> 9) But there is an another problem.The axis2 service
> >>> corresponding to
> >>> > a
> >>> > >>>> deployed process does not appear on all nodes of the cluster.
> >>> That is
> >>> > >>>> because each server add du which is deployed by it to the
> process
> >>> > >>>> store.That is why I had use loadAll() when masters are changing.
> >>> How
> >>> > to
> >>> > >>>> solve this?
> >>> > >>>>
> >>> > >>>
> >>> > >>>     I do appriciate your efforts in understanding the
> >>> implmentation and
> >>> > >>> changes that need to be done. You are bang on it.
> >>> > >>>     fireEvent(..) is the method that triggers process activation
> >>> and
> >>> > >>> necessary service creation.
> >>> > >>>
> >>> > >>>
> >>> > >>>     But with this given apporach from steps 6 to 8, ODE cannot
> have
> >>> > >>> atleast 2 Active servers for Load balancing. You are
> concentrating
> >>> on
> >>> > only
> >>> > >>> one active node that will do deployments and cater to process
> >>> > invocations.
> >>> > >>>     We should also think about scaling ODE to multiple servers to
> >>> > handle
> >>> > >>> load.
> >>> > >>>
> >>> > >>>     What do you think.
> >>> > >>>
> >>> > >>>
> >>> > >>>> Thank you,
> >>> > >>>> Sudharma
> >>> > >>>>
> >>> > >>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com>
> >>> wrote:
> >>> > >>>>
> >>> > >>>> > Sudharma,
> >>> > >>>> >
> >>> > >>>> > Any updates?
> >>> > >>>> >
> >>> > >>>> > regards,
> >>> > >>>> > sathwik
> >>> > >>>> >
> >>> > >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <
> >>> sathwik.bp@gmail.com>
> >>> > >>>> wrote:
> >>> > >>>> >
> >>> > >>>> > > Sudharma,
> >>> > >>>> > >
> >>> > >>>> > > Can you elaborate on your option 1).
> >>> > >>>> > >
> >>> > >>>> > > Response to your option 2).
> >>> > >>>> > >
> >>> > >>>> > >     Process Store is the component that handles process
> >>> metadata,
> >>> > >>>> > > compilation and deployment in ODE. Integration layers in ODE
> >>> > >>>> (Axis2, JBI)
> >>> > >>>> > > use the process store.
> >>> > >>>> > >     Future implementations of IL for ODE will also use the
> >>> process
> >>> > >>>> store.
> >>> > >>>> > > We should not be thinking of moving the process store
> >>> > functionality
> >>> > >>>> to
> >>> > >>>> > the
> >>> > >>>> > > integration layers.
> >>> > >>>> > >
> >>> > >>>> > >
> >>> > >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> >>> > >>>> > > suba.11@cse.mrt.ac.lk> wrote:
> >>> > >>>> > >
> >>> > >>>> > >> Hi,
> >>> > >>>> > >>
> >>> > >>>> > >> I understood the problem within dynamic master/slave
> >>> > >>>> configuration. In
> >>> > >>>> > my
> >>> > >>>> > >> approach, when a deployment request is routed to a slave
> node
> >>> > >>>> there will
> >>> > >>>> > >> not be a deployment. I suggest two options to avoid it.
> >>> > >>>> > >> 1) Have static master/slave configuration only for deploy
> >>> process
> >>> > >>>> > >>
> >>> > >>>> > > 2) Modify the deployment web service to complie and verify
> the
> >>> > >>>> process
> >>> > >>>> > and
> >>> > >>>> > >> then copy it to the deploy folder irrespective of whether
> >>> its a
> >>> > >>>> master
> >>> > >>>> > or
> >>> > >>>> > >> slave, then deployment poller should take care of the
> >>> deployment
> >>> > >>>> > >>
> >>> > >>>> > >>
> >>> > >>>> > >
> >>> > >>>> > >>
> >>> > >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sathwik.bp@gmail.com
> >
> >>> > wrote:
> >>> > >>>> > >>
> >>> > >>>> > >> > Sudharma,
> >>> > >>>> > >> >
> >>> > >>>> > >> > We definitely need a master/slave in the hazelcast
> cluster.
> >>> > This
> >>> > >>>> is
> >>> > >>>> > >> > probably needed for the job migration in the Scheduler to
> >>> > >>>> migrate the
> >>> > >>>> > >> jobs
> >>> > >>>> > >> > associated with a down node. Let hold on this topic for
> >>> future
> >>> > >>>> > >> discussion.
> >>> > >>>> > >> >
> >>> > >>>> > >> > Going by the explanation where the master/slave nodes
> have
> >>> > >>>> certain
> >>> > >>>> > >> > predefined tasks to perform is perfectly fine.
> >>> > >>>> > >> >
> >>> > >>>> > >> > I have this scenario,
> >>> > >>>> > >> >
> >>> > >>>> > >> > I am using HAProxy as my load balancer and configured 3
> >>> nodes
> >>> > in
> >>> > >>>> the
> >>> > >>>> > >> > cluster.
> >>> > >>>> > >> >
> >>> > >>>> > >> > Node1 - Active
> >>> > >>>> > >> > Node2 - Active
> >>> > >>>> > >> > Node3 - Backup
> >>> > >>>> > >> >
> >>> > >>>> > >> > Load balancing algorithm: RoundRobin
> >>> > >>>> > >> >
> >>> > >>>> > >> > A Backup node (Node3) is one which the load balancer will
> >>> not
> >>> > >>>> route
> >>> > >>>> > >> > requests to, until one of the Active node i.e either
> Node1
> >>> or
> >>> > >>>> Node2
> >>> > >>>> > has
> >>> > >>>> > >> > gone down.
> >>> > >>>> > >> >
> >>> > >>>> > >> > All these 3 nodes are also part of the hazelcast cluster
> as
> >>> > well.
> >>> > >>>> > >> >
> >>> > >>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
> >>> > >>>> leader/master
> >>> > >>>> > >> and
> >>> > >>>> > >> > Node2,Node3 as slaves.
> >>> > >>>> > >> >
> >>> > >>>> > >> > I initiate the deploy operation on the
> DeploymentWebService
> >>> > >>>> which the
> >>> > >>>> > >> load
> >>> > >>>> > >> > balancer routes it to one of the Active nodes in the
> >>> cluster,
> >>> > >>>> lets say
> >>> > >>>> > >> it's
> >>> > >>>> > >> > the Node1. Since Node1 is also the master in the
> hazelcast
> >>> > >>>> cluster,
> >>> > >>>> > >> > deployment is a success.
> >>> > >>>> > >> >
> >>> > >>>> > >> > I initiate another deploy operation on the
> >>> DeploymentWebService
> >>> > >>>> which
> >>> > >>>> > >> the
> >>> > >>>> > >> > load balancer routes it to the next active node which is
> >>> Node2.
> >>> > >>>> Since
> >>> > >>>> > >> Node2
> >>> > >>>> > >> > is a slave in the Hazelcast cluster, What happens to the
> >>> > >>>> deployment?
> >>> > >>>> > >> >
> >>> > >>>> > >> > regards,
> >>> > >>>> > >> > sathwik
> >>> > >>>> > >> >
> >>> > >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> >>> > >>>> > >> > suba.11@cse.mrt.ac.lk
> >>> > >>>> > >> > > wrote:
> >>> > >>>> > >> >
> >>> > >>>> > >> > > Hi,
> >>> > >>>> > >> > >
> >>> > >>>> > >> > > I will explain my approach as much as possible. The
> >>> oldest
> >>> > >>>> node in
> >>> > >>>> > the
> >>> > >>>> > >> > > hazelcast cluster is elected as the master node. In the
> >>> > >>>> failure of
> >>> > >>>> > the
> >>> > >>>> > >> > > master node, next oldest node will be elected as the
> >>> master
> >>> > >>>> node.
> >>> > >>>> > This
> >>> > >>>> > >> > > master-slave configuration is just for deployment. When
> >>> the
> >>> > >>>> > hazelcast
> >>> > >>>> > >> > > cluster elected the master node, that node becomes a
> >>> master
> >>> > >>>> node for
> >>> > >>>> > >> > > deploying process. So it will do the deploying
> >>> artifacts. If
> >>> > >>>> you
> >>> > >>>> > want
> >>> > >>>> > >> to
> >>> > >>>> > >> > > get the idea of electing master node please refer the
> >>> code
> >>> > >>>> which I
> >>> > >>>> > >> have
> >>> > >>>> > >> > > located in the github. (
> >>> > >>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
> >>> > >>>> > >> > >
> >>> > >>>> > >> > > I identified separated actions which should be followed
> >>> by
> >>> > the
> >>> > >>>> > master
> >>> > >>>> > >> and
> >>> > >>>> > >> > > salve nodes.
> >>> > >>>> > >> > > Actions which are followed by master node only
> >>> > >>>> > >> > > 1) create deployment unit
> >>> > >>>> > >> > > 2) set the version nu to deployment unit
> >>> > >>>> > >> > > 3) compile deployment unit
> >>> > >>>> > >> > > 4) scan deployment unit
> >>> > >>>> > >> > > 5) retire previous versions
> >>> > >>>> > >> > > Master node and slave nodes should create _processes
> >>> which
> >>> > >>>> stores
> >>> > >>>> > >> > > ProcessConfImpl
> >>> > >>>> > >> > > Only master node will write the version nu to database,
> >>> > create
> >>> > >>>> > >> .deployed
> >>> > >>>> > >> > > file
> >>> > >>>> > >> > >
> >>> > >>>> > >> > > So there are some actions which should be followed only
> >>> by
> >>> > >>>> master
> >>> > >>>> > node
> >>> > >>>> > >> > > while other actions should be followed by all the
> >>> nodes.The
> >>> > >>>> idea of
> >>> > >>>> > >> > having
> >>> > >>>> > >> > > a master node is deploying artifacts and avoid others
> >>> from
> >>> > >>>> writing
> >>> > >>>> > the
> >>> > >>>> > >> > > version nu to database.
> >>> > >>>> > >> > > Whether a node is active or passive, all nodes should
> do
> >>> the
> >>> > >>>> > >> > > deployment.Master
> >>> > >>>> > >> > > and slaves will follow necessary actions as in above.
> >>> > >>>> > >> > >
> >>> > >>>> > >> > >
> >>> > >>>> > >> > >
> >>> > >>>> > >> > >
> >>> > >>>> > >> > >
> >>> > >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <
> >>> sathwik.bp@gmail.com>
> >>> > >>>> wrote:
> >>> > >>>> > >> > >
> >>> > >>>> > >> > > > Nandika,
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > I very well understand what you have put across, but
> >>> it's
> >>> > >>>> > secondary
> >>> > >>>> > >> to
> >>> > >>>> > >> > me
> >>> > >>>> > >> > > > now.
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > Sudharma,
> >>> > >>>> > >> > > > My primary concern is to understand at a high level
> the
> >>> > >>>> deployment
> >>> > >>>> > >> > > > architecture and how would master-slave configuration
> >>> fit
> >>> > >>>> in. Are
> >>> > >>>> > >> there
> >>> > >>>> > >> > > any
> >>> > >>>> > >> > > > restrictions imposed by the in-progress design?
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > Firstly, how would ODE process deployment work under
> >>> these
> >>> > >>>> cluster
> >>> > >>>> > >> > > > configurations?
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > Sample Cluster configurations: A load balancer is
> >>> > >>>> frontending the
> >>> > >>>> > >> > > servers.
> >>> > >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> >>> > >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> >>> > >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either
> >>> in
> >>> > >>>> Active or
> >>> > >>>> > >> > > Passive.
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > regards,
> >>> > >>>> > >> > > > sathwik
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana
> <
> >>> > >>>> > >> > jayawark@gmail.com
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > wrote:
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > > > > Hi Sathwik,
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > > According to my understanding, in the clustering
> >>> > scenario,
> >>> > >>>> the
> >>> > >>>> > >> master
> >>> > >>>> > >> > > > node
> >>> > >>>> > >> > > > > should perform all the deployment actions and the
> >>> slave
> >>> > >>>> nodes
> >>> > >>>> > also
> >>> > >>>> > >> > need
> >>> > >>>> > >> > > > to
> >>> > >>>> > >> > > > > perform some deployment actions. For example, the
> >>> slave
> >>> > >>>> nodes
> >>> > >>>> > also
> >>> > >>>> > >> > > should
> >>> > >>>> > >> > > > > handle the process ACTIVATED event so that the
> >>> process
> >>> > >>>> > >> configuration
> >>> > >>>> > >> > is
> >>> > >>>> > >> > > > > added to the engine and necessary web services are
> >>> > created
> >>> > >>>> so
> >>> > >>>> > that
> >>> > >>>> > >> > when
> >>> > >>>> > >> > > > the
> >>> > >>>> > >> > > > > load balancer send requests to any node in the
> >>> cluster,
> >>> > it
> >>> > >>>> is
> >>> > >>>> > >> ready
> >>> > >>>> > >> > to
> >>> > >>>> > >> > > > > accept those requests.
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > > Regards
> >>> > >>>> > >> > > > > Nandika
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> >>> > >>>> > >> sathwik.bp@gmail.com>
> >>> > >>>> > >> > > > > wrote:
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > > > > Sudharma,
> >>> > >>>> > >> > > > > >
> >>> > >>>> > >> > > > > > Where are you going to configure the
> >>> master-slaves, is
> >>> > >>>> it in
> >>> > >>>> > the
> >>> > >>>> > >> > web
> >>> > >>>> > >> > > > > > application level or at the load balancer?
> >>> > >>>> > >> > > > > >
> >>> > >>>> > >> > > > > > regards,
> >>> > >>>> > >> > > > > > sathwik
> >>> > >>>> > >> > > > > >
> >>> > >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma
> >>> subasinghe <
> >>> > >>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
> >>> > >>>> > >> > > > > > wrote:
> >>> > >>>> > >> > > > > >
> >>> > >>>> > >> > > > > > > Hi Tammo,
> >>> > >>>> > >> > > > > > >
> >>> > >>>> > >> > > > > > > Can you suggest the best method from these to
> >>> > >>>> implement? As
> >>> > >>>> > >> > first I
> >>> > >>>> > >> > > > > > > suggested the master-slaves scenario I think it
> >>> is
> >>> > >>>> easy to
> >>> > >>>> > >> > > implement
> >>> > >>>> > >> > > > > than
> >>> > >>>> > >> > > > > > > distributed lock scenario. However if you can
> >>> suggest
> >>> > >>>> one
> >>> > >>>> > from
> >>> > >>>> > >> > > these
> >>> > >>>> > >> > > > > two,
> >>> > >>>> > >> > > > > > > then I can think about it.
> >>> > >>>> > >> > > > > > >
> >>> > >>>> > >> > > > > > > Thank you
> >>> > >>>> > >> > > > > > >
> >>> > >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
> >>> > >>>> sathwik.bp@gmail.com>
> >>> > >>>> > >> > wrote:
> >>> > >>>> > >> > > > > > >
> >>> > >>>> > >> > > > > > > > With respect to the hotdeployment,
> >>> > >>>> > >> > > > > > > >
> >>> > >>>> > >> > > > > > > > We can drop the deployment archive onto the
> >>> > >>>> deployment
> >>> > >>>> > >> folder.
> >>> > >>>> > >> > > > Since
> >>> > >>>> > >> > > > > > the
> >>> > >>>> > >> > > > > > > > DeploymentPoller are acquiring the
> distributed
> >>> lock
> >>> > >>>> for
> >>> > >>>> > the
> >>> > >>>> > >> > > > > > > DeploymentUnit,
> >>> > >>>> > >> > > > > > > > only one of the nodes will get the lock and
> >>> > initiate
> >>> > >>>> the
> >>> > >>>> > >> > > > deployment.
> >>> > >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
> >>> > >>>> acquiring
> >>> > >>>> > the
> >>> > >>>> > >> > lock
> >>> > >>>> > >> > > > and
> >>> > >>>> > >> > > > > > > hence
> >>> > >>>> > >> > > > > > > > will silently ignore it.
> >>> > >>>> > >> > > > > > > >
> >>> > >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B
> P <
> >>> > >>>> > >> > > > sathwik.bp@gmail.com>
> >>> > >>>> > >> > > > > > > > wrote:
> >>> > >>>> > >> > > > > > > >
> >>> > >>>> > >> > > > > > > > > Hi Tammo,
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > The distributed lock acquisition on the
> >>> > >>>> DeploymentUnit
> >>> > >>>> > >> should
> >>> > >>>> > >> > > be
> >>> > >>>> > >> > > > > > added
> >>> > >>>> > >> > > > > > > to
> >>> > >>>> > >> > > > > > > > > both DeploymentWebService and
> >>> DeploymentPoller.
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > When a deployment operation is initiated
> >>> through
> >>> > >>>> the
> >>> > >>>> > >> > > > > > > > DeploymentWebService,
> >>> > >>>> > >> > > > > > > > > The load balancer routes it to any of the
> >>> > available
> >>> > >>>> > nodes.
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > On the routed node, the
> DeploymentWebService
> >>> > >>>> acquires
> >>> > >>>> > the
> >>> > >>>> > >> > > > > Distributed
> >>> > >>>> > >> > > > > > > > > lock. On the remaining nodes the
> >>> DeploymentPoller
> >>> > >>>> will
> >>> > >>>> > >> try to
> >>> > >>>> > >> > > > > acquire
> >>> > >>>> > >> > > > > > > the
> >>> > >>>> > >> > > > > > > > > distributed lock and will not get it and
> >>> hence
> >>> > will
> >>> > >>>> > >> silently
> >>> > >>>> > >> > > > ignore
> >>> > >>>> > >> > > > > > it.
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > Once the routed node completes the
> >>> deployment, it
> >>> > >>>> will
> >>> > >>>> > >> > release
> >>> > >>>> > >> > > > the
> >>> > >>>> > >> > > > > > > lock.
> >>> > >>>> > >> > > > > > > > > This way we don't have to stall the
> >>> > >>>> DeploymentPoller in
> >>> > >>>> > >> other
> >>> > >>>> > >> > > > > nodes.
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > Does it answer the concerns?
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > Now, if we give the responsibility of
> >>> identifying
> >>> > >>>> the
> >>> > >>>> > >> master
> >>> > >>>> > >> > > node
> >>> > >>>> > >> > > > > to
> >>> > >>>> > >> > > > > > > the
> >>> > >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the
> >>> load
> >>> > >>>> balancer
> >>> > >>>> > to
> >>> > >>>> > >> > > change
> >>> > >>>> > >> > > > > > it's
> >>> > >>>> > >> > > > > > > > > configuration about the master node?
> >>> > >>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
> >>> > >>>> > >> > > > > > > > > node1 -master
> >>> > >>>> > >> > > > > > > > > node2 - slave
> >>> > >>>> > >> > > > > > > > > node3 - slave
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2
> as
> >>> > >>>> master
> >>> > >>>> > node,
> >>> > >>>> > >> > but
> >>> > >>>> > >> > > > > > > hazelcast
> >>> > >>>> > >> > > > > > > > > might promote Node3 as master node. They
> are
> >>> out
> >>> > of
> >>> > >>>> > sync.
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > Is this argument valid?
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > regards,
> >>> > >>>> > >> > > > > > > > > sathwik
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van
> >>> > Lessen <
> >>> > >>>> > >> > > > > > > tvanlessen@gmail.com>
> >>> > >>>> > >> > > > > > > > > wrote:
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > >> Hi Sudharma,
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >> what do you expect from the "other nodes
> >>> > >>>> deployment"?
> >>> > >>>> > >> > > > Compilation
> >>> > >>>> > >> > > > > is
> >>> > >>>> > >> > > > > > > not
> >>> > >>>> > >> > > > > > > > >> needed since the CBP file is written to
> the
> >>> > >>>> (shared)
> >>> > >>>> > FS.
> >>> > >>>> > >> > > > > > Registration
> >>> > >>>> > >> > > > > > > is
> >>> > >>>> > >> > > > > > > > >> also not needed, since it is done via the
> >>> shared
> >>> > >>>> > >> database.
> >>> > >>>> > >> > So
> >>> > >>>> > >> > > > the
> >>> > >>>> > >> > > > > > only
> >>> > >>>> > >> > > > > > > > >> thing that might be needed is to tell the
> >>> engine
> >>> > >>>> that
> >>> > >>>> > >> there
> >>> > >>>> > >> > > is a
> >>> > >>>> > >> > > > > new
> >>> > >>>> > >> > > > > > > > >> deployment. I'd need to check that. If
> this
> >>> is
> >>> > >>>> needed,
> >>> > >>>> > I
> >>> > >>>> > >> > > revert
> >>> > >>>> > >> > > > my
> >>> > >>>> > >> > > > > > > last
> >>> > >>>> > >> > > > > > > > >> statement, then it is perhaps better to
> just
> >>> > send
> >>> > >>>> an
> >>> > >>>> > >> event
> >>> > >>>> > >> > > over
> >>> > >>>> > >> > > > > > > > Hazelcast
> >>> > >>>> > >> > > > > > > > >> to all nodes that the deployment has
> >>> changed.
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >> Best,
> >>> > >>>> > >> > > > > > > > >>   Tammo
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
> >>> > >>>> subasinghe <
> >>> > >>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
> >>> > >>>> > >> > > > > > > > >> > wrote:
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >> > Hi Tammo,
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >> > The master node writes meta data. But
> >>> runtime
> >>> > >>>> > >> information
> >>> > >>>> > >> > > must
> >>> > >>>> > >> > > > > be
> >>> > >>>> > >> > > > > > > > >> available
> >>> > >>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared,
> >>> all
> >>> > >>>> nodes
> >>> > >>>> > will
> >>> > >>>> > >> > see
> >>> > >>>> > >> > > > the
> >>> > >>>> > >> > > > > > > > >> > availability of a new process. My idea
> is
> >>> for
> >>> > >>>> master
> >>> > >>>> > >> node
> >>> > >>>> > >> > to
> >>> > >>>> > >> > > > > write
> >>> > >>>> > >> > > > > > > the
> >>> > >>>> > >> > > > > > > > >> meta
> >>> > >>>> > >> > > > > > > > >> > data and other nodes to just read the
> meta
> >>> > data
> >>> > >>>> and
> >>> > >>>> > >> load
> >>> > >>>> > >> > > > > > process.So
> >>> > >>>> > >> > > > > > > we
> >>> > >>>> > >> > > > > > > > >> need
> >>> > >>>> > >> > > > > > > > >> > a small delay between master node
> >>> deployment
> >>> > and
> >>> > >>>> > other
> >>> > >>>> > >> > nodes
> >>> > >>>> > >> > > > > > > > deployment.
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >> > Is there anyway to set the delay between
> >>> > master
> >>> > >>>> node
> >>> > >>>> > >> and
> >>> > >>>> > >> > > > slaves
> >>> > >>>> > >> > > > > > > until
> >>> > >>>> > >> > > > > > > > >> > master node finish the deployment?
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >> > Thank you
> >>> > >>>> > >> > > > > > > > >> > Sudharma
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van
> Lessen
> >>> <
> >>> > >>>> > >> > > > tvanlessen@gmail.com
> >>> > >>>> > >> > > > > >
> >>> > >>>> > >> > > > > > > > wrote:
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >> > > Hi Sathwik,
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM,
> >>> Sathwik B
> >>> > P <
> >>> > >>>> > >> > > > > > > sathwik.bp@gmail.com>
> >>> > >>>> > >> > > > > > > > >> > wrote:
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
> >>> > >>>> > >> > > > > > > > >> > > >
> >>> > >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is
> >>> the
> >>> > >>>> master
> >>> > >>>> > >> node
> >>> > >>>> > >> > in
> >>> > >>>> > >> > > > the
> >>> > >>>> > >> > > > > > > > cluster?
> >>> > >>>> > >> > > > > > > > >> > > >
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > I think the easiest approach is to
> >>> always
> >>> > >>>> elect the
> >>> > >>>> > >> > oldest
> >>> > >>>> > >> > > > > node
> >>> > >>>> > >> > > > > > in
> >>> > >>>> > >> > > > > > > > the
> >>> > >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK
> >>> Hazelcast
> >>> > can
> >>> > >>>> > easily
> >>> > >>>> > >> > asked
> >>> > >>>> > >> > > > for
> >>> > >>>> > >> > > > > > > this
> >>> > >>>> > >> > > > > > > > >> > > information.
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the
> >>> Deployment
> >>> > >>>> Pollers
> >>> > >>>> > in
> >>> > >>>> > >> > the
> >>> > >>>> > >> > > > > slave
> >>> > >>>> > >> > > > > > > > nodes?
> >>> > >>>> > >> > > > > > > > >> > > >
> >>> > >>>> > >> > > > > > > > >> > > >
> >>> > >>>> > >> > > > > > > > >> > > Absolutely.
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > Suggestion:
> >>> > >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
> >>> > >>>> Master-SLaves.
> >>> > >>>> > Why
> >>> > >>>> > >> > not
> >>> > >>>> > >> > > > give
> >>> > >>>> > >> > > > > > > every
> >>> > >>>> > >> > > > > > > > >> node
> >>> > >>>> > >> > > > > > > > >> > > in
> >>> > >>>> > >> > > > > > > > >> > > > the cluster the same status
> >>> > (Active-Active).
> >>> > >>>> > >> > > > > > > > >> > > >
> >>> > >>>> > >> > > > > > > > >> > > > When a new deployment is made, the
> >>> load
> >>> > >>>> balancer
> >>> > >>>> > >> can
> >>> > >>>> > >> > > push
> >>> > >>>> > >> > > > it
> >>> > >>>> > >> > > > > > to
> >>> > >>>> > >> > > > > > > > any
> >>> > >>>> > >> > > > > > > > >> of
> >>> > >>>> > >> > > > > > > > >> > > the
> >>> > >>>> > >> > > > > > > > >> > > > available nodes. That node will
> >>> probably
> >>> > >>>> acquire
> >>> > >>>> > a
> >>> > >>>> > >> > > > > distributed
> >>> > >>>> > >> > > > > > > > lock
> >>> > >>>> > >> > > > > > > > >> on
> >>> > >>>> > >> > > > > > > > >> > > the
> >>> > >>>> > >> > > > > > > > >> > > > deployment unit and acts as master
> for
> >>> > that
> >>> > >>>> > >> > deployment.
> >>> > >>>> > >> > > > This
> >>> > >>>> > >> > > > > > > > ensures
> >>> > >>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
> >>> > >>>> Probably no
> >>> > >>>> > >> static
> >>> > >>>> > >> > > > > > > > >> configuration of
> >>> > >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer
> nor
> >>> in
> >>> > the
> >>> > >>>> > >> > hazelcast.
> >>> > >>>> > >> > > > > > > > >> > > >
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > But this would not allow to have the
> >>> > >>>> hotdeployment
> >>> > >>>> > >> via
> >>> > >>>> > >> > > > > > filesystem
> >>> > >>>> > >> > > > > > > > >> still
> >>> > >>>> > >> > > > > > > > >> > > enabled, right?
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > Best,
> >>> > >>>> > >> > > > > > > > >> > >   Tammo
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> > > --
> >>> > >>>> > >> > > > > > > > >> > > Tammo van Lessen -
> http://www.taval.de
> >>> > >>>> > >> > > > > > > > >> > >
> >>> > >>>> > >> > > > > > > > >> >
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >> --
> >>> > >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> >>> > >>>> > >> > > > > > > > >>
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > > >
> >>> > >>>> > >> > > > > > > >
> >>> > >>>> > >> > > > > > >
> >>> > >>>> > >> > > > > >
> >>> > >>>> > >> > > > >
> >>> > >>>> > >> > > >
> >>> > >>>> > >> > >
> >>> > >>>> > >> >
> >>> > >>>> > >>
> >>> > >>>> > >
> >>> > >>>> > >
> >>> > >>>> >
> >>> > >>>>
> >>> > >>>
> >>> > >>>
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Sathwik,

I modified the code as you explained. You can go through the code by
following link.
https://github.com/Subasinghe/ode/tree/ODECluster

It would be helpful if you can provide a feedback on this.

Thank you.

On 10 June 2015 at 12:39, sudharma subasinghe <su...@cse.mrt.ac.lk> wrote:

> Hi Sathwik,
>
> I tried to implement your logic also. Following is the link for committed
> code upto now.
> https://github.com/Subasinghe/ode/tree/ODECluster
>
> I used info msgs to debugging as it can be observed easily in console.
> When releasing the lock acquired by poller or web service the state is set
> to false as in my code. But sometimes it is set to true and at that time
> second server's value is also true.  Following is the related log for each
> server.
>
> Server 1
> 02:12:02,957 INFO  [DeploymentPoller] Trying to access the lock for
> MagicSession
> 02:12:02,962 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value for
> MagicSession file after locking: true
> 02:12:03,838 INFO  [BpelServerImpl] Registered process {
> http://ode/bpel/unit-test}MagicSessionMain-3.
> 02:12:04,015 INFO  [BpelServerImpl] Registered process {
> http://ode/bpel/responder}MagicSessionResponder-3.
> 02:12:04,015 INFO  [DeploymentPoller] Deployment of artifact MagicSession
> successful: [{http://ode/bpel/unit-test}MagicSessionMain-3, {
> http://ode/bpel/responder}MagicSessionResponder-3]
> 02:12:04,015 INFO  [DeploymentPoller] Trying to release the lock for
> MagicSession
> 02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value for
> MagicSession file after unlocking: true
>
> Server 2
> 02:12:03,119 INFO  [DeploymentPoller] Trying to access the lock for
> MagicSession
> 02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value for
> MagicSession file after locking: true
> 02:12:04,017 INFO  [DeploymentPoller] Trying to release the lock for
> MagicSession
> 02:12:04,019 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value for
> MagicSession file after unlocking: false
>
> Is this possible? But there were not any conflicts while deploying. It
> worked perfectly.
>
> Thank you
>
>
> On 7 June 2015 at 11:10, sudharma subasinghe <su...@cse.mrt.ac.lk>
> wrote:
>
>> Hi Sathwik,
>>
>> I will concern your approach. Thank you for your effort.
>>
>> On 6 June 2015 at 18:43, Sathwik B P <sa...@gmail.com> wrote:
>>
>>> I have attached to the JIRA
>>> https://issues.apache.org/jira/browse/ODE-563
>>>
>>> On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <
>>> suba.11@cse.mrt.ac.lk>
>>> wrote:
>>>
>>> > Hi Sathwik,
>>> >
>>> > I can't find the attached document. Please kind be enough to resend it.
>>> >
>>> > On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:
>>> >
>>> > > Refer this one as I have corrected the numbering of steps.
>>> > >
>>> > > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com>
>>> > wrote:
>>> > >
>>> > >> Hi Sudharma/Tammo,
>>> > >>
>>> > >> Kindly review attached document on my proposed approach. Let me
>>> know if
>>> > >> you have any concerns or doubts on it.
>>> > >>
>>> > >> regards,
>>> > >> sathwik
>>> > >>
>>> > >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com>
>>> > wrote:
>>> > >>
>>> > >>> find response inline
>>> > >>>
>>> > >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
>>> > >>> suba.11@cse.mrt.ac.lk> wrote:
>>> > >>>
>>> > >>>> Hi,
>>> > >>>>
>>> > >>>> Sorry for the late reply. I was trying to achieve a proper
>>> solution.
>>> > >>>> Following is my approach.
>>> > >>>>
>>> > >>>> 1) I elected master node for deploying purpose. So when a master
>>> node
>>> > >>>> goes
>>> > >>>> down hazelcast will elect the next oldest node for master.
>>> > >>>>
>>> > >>>
>>> > >>>     Perfect
>>> > >>>
>>> > >>>
>>> > >>>> 2) ODEServer can identify whether clustering is enabled or not by
>>> > >>>> getting
>>> > >>>> the property value in ode-axis2.properties file. So I introduced
>>> new
>>> > >>>> property called "ode-axis2.hazelcast.clustering.enabled". If
>>> there is
>>> > no
>>> > >>>> clustering enabled server will work as it is. If clustering is
>>> > enabled,
>>> > >>>> cluster will be initialized.
>>> > >>>>
>>> > >>>
>>> > >>>     Perfect
>>> > >>>
>>> > >>>
>>> > >>>> 3) In manual deployment its responsibility is taken by the
>>> > >>>> DeploymentPoller. So I give the deployment capability to master
>>> node,s
>>> > >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to
>>> true.So
>>> > >>>> others will not be able to go into check() method in
>>> DeploymentPoller.
>>> > >>>> So
>>> > >>>> deployment will be done by only the master node.
>>> > >>>>
>>> > >>>
>>> > >>>     Perfect
>>> > >>>
>>> > >>>
>>> > >>>>
>>> > >>>> 4) In DeploymentWebService, I had to consider few cases.If the
>>> deploy
>>> > >>>> request goes to the master node, it will deploy the process
>>> through
>>> > web
>>> > >>>> service.Others pollers will not go into check() method as they
>>> are not
>>> > >>>> masters. So master can continue without any involvement of others.
>>> > >>>>
>>> > >>>>
>>> > >>>     Perfect
>>> > >>>
>>> > >>>
>>> > >>>> 5) If the deploy request goes to a slave node, it will do up to
>>> file
>>> > >>>> creation in the file system.Slave will be stopped at that point.
>>> As
>>> > only
>>> > >>>> master poller is checking, master can continue from created files
>>> in
>>> > the
>>> > >>>> file system.
>>> > >>>>
>>> > >>>
>>> > >>>     DeploymentWebService provides synchronous operations. The
>>> status of
>>> > >>> the operation should be communicated to the calling client in the
>>> same
>>> > call.
>>> > >>>     DeploymentPoller is a backend thread that goes over each and
>>> every
>>> > >>> directory under the Deployment directory checking for any changes
>>> to
>>> > >>> deploy.xml in existing processes and deploy newly added processes.
>>> This
>>> > >>> process is  sequential and time consuming. As the process
>>> directories
>>> > >>> grows, so does the time taken for execution of the thread.
>>> > >>>
>>> > >>>     Since the request is on a slave node and the processing is
>>> done on
>>> > >>> master node, how do you check for the completion of the
>>> > >>> deployment/undeployment of processes and respond back to the client
>>> > since
>>> > >>> the web service call is a synchronous operation. As
>>> DeploymentPoller is
>>> > >>> taking a lot of time in processing, your request will time out
>>> right.
>>> > >>>
>>> > >>>
>>> > >>>>
>>> > >>>> 6) But there was problem with _deploymentUnits in
>>> ProcessStoreImpl.
>>> > Each
>>> > >>>> _deploymentUnits stores only what its server has deployed. So
>>> think,
>>> > >>>> that a
>>> > >>>> master node goes down another master node appears.But its
>>> > >>>> __deploymentUnits
>>> > >>>> does not have dus which has deployed by the earlier master node.
>>> Hence
>>> > >>>> it
>>> > >>>> will not be able retire earlier version of the process which is
>>> > >>>> deployed by
>>> > >>>> previous master. So there will two process which are in "ACTIVE"
>>> state
>>> > >>>
>>> > >>>
>>> > >>>> 7) To avoid this, I add the ODEServer as an Observer to check
>>> when a
>>> > new
>>> > >>>> master is electing, then load all the deployment units out of the
>>> > >>>> store. So
>>> > >>>> new master node can have all the dus and can retire appropriate
>>> > version.
>>> > >>>> Usually loadAll() is called at the server start-up time. But
>>> there is
>>> > no
>>> > >>>> other way to solve this. I tried to use Hazelcast IMap to store
>>> all
>>> > dus
>>> > >>>> among all nodes. But it wasn't success as du is not serializable
>>> > >>>> object.
>>> > >>>>
>>> > >>>
>>> > >>>> 8) I figured out that we do not need send cluster message to
>>> others as
>>> > >>>> all
>>> > >>>> the dus' data are persisted to the shared DB. So each node can
>>> take
>>> > the
>>> > >>>> du
>>> > >>>> and retrieve necessary data using already implemented methods in
>>> > Process
>>> > >>>> Store.
>>> > >>>>
>>> > >>>> 9) But there is an another problem.The axis2 service
>>> corresponding to
>>> > a
>>> > >>>> deployed process does not appear on all nodes of the cluster.
>>> That is
>>> > >>>> because each server add du which is deployed by it to the process
>>> > >>>> store.That is why I had use loadAll() when masters are changing.
>>> How
>>> > to
>>> > >>>> solve this?
>>> > >>>>
>>> > >>>
>>> > >>>     I do appriciate your efforts in understanding the
>>> implmentation and
>>> > >>> changes that need to be done. You are bang on it.
>>> > >>>     fireEvent(..) is the method that triggers process activation
>>> and
>>> > >>> necessary service creation.
>>> > >>>
>>> > >>>
>>> > >>>     But with this given apporach from steps 6 to 8, ODE cannot have
>>> > >>> atleast 2 Active servers for Load balancing. You are concentrating
>>> on
>>> > only
>>> > >>> one active node that will do deployments and cater to process
>>> > invocations.
>>> > >>>     We should also think about scaling ODE to multiple servers to
>>> > handle
>>> > >>> load.
>>> > >>>
>>> > >>>     What do you think.
>>> > >>>
>>> > >>>
>>> > >>>> Thank you,
>>> > >>>> Sudharma
>>> > >>>>
>>> > >>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com>
>>> wrote:
>>> > >>>>
>>> > >>>> > Sudharma,
>>> > >>>> >
>>> > >>>> > Any updates?
>>> > >>>> >
>>> > >>>> > regards,
>>> > >>>> > sathwik
>>> > >>>> >
>>> > >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <
>>> sathwik.bp@gmail.com>
>>> > >>>> wrote:
>>> > >>>> >
>>> > >>>> > > Sudharma,
>>> > >>>> > >
>>> > >>>> > > Can you elaborate on your option 1).
>>> > >>>> > >
>>> > >>>> > > Response to your option 2).
>>> > >>>> > >
>>> > >>>> > >     Process Store is the component that handles process
>>> metadata,
>>> > >>>> > > compilation and deployment in ODE. Integration layers in ODE
>>> > >>>> (Axis2, JBI)
>>> > >>>> > > use the process store.
>>> > >>>> > >     Future implementations of IL for ODE will also use the
>>> process
>>> > >>>> store.
>>> > >>>> > > We should not be thinking of moving the process store
>>> > functionality
>>> > >>>> to
>>> > >>>> > the
>>> > >>>> > > integration layers.
>>> > >>>> > >
>>> > >>>> > >
>>> > >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
>>> > >>>> > > suba.11@cse.mrt.ac.lk> wrote:
>>> > >>>> > >
>>> > >>>> > >> Hi,
>>> > >>>> > >>
>>> > >>>> > >> I understood the problem within dynamic master/slave
>>> > >>>> configuration. In
>>> > >>>> > my
>>> > >>>> > >> approach, when a deployment request is routed to a slave node
>>> > >>>> there will
>>> > >>>> > >> not be a deployment. I suggest two options to avoid it.
>>> > >>>> > >> 1) Have static master/slave configuration only for deploy
>>> process
>>> > >>>> > >>
>>> > >>>> > > 2) Modify the deployment web service to complie and verify the
>>> > >>>> process
>>> > >>>> > and
>>> > >>>> > >> then copy it to the deploy folder irrespective of whether
>>> its a
>>> > >>>> master
>>> > >>>> > or
>>> > >>>> > >> slave, then deployment poller should take care of the
>>> deployment
>>> > >>>> > >>
>>> > >>>> > >>
>>> > >>>> > >
>>> > >>>> > >>
>>> > >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com>
>>> > wrote:
>>> > >>>> > >>
>>> > >>>> > >> > Sudharma,
>>> > >>>> > >> >
>>> > >>>> > >> > We definitely need a master/slave in the hazelcast cluster.
>>> > This
>>> > >>>> is
>>> > >>>> > >> > probably needed for the job migration in the Scheduler to
>>> > >>>> migrate the
>>> > >>>> > >> jobs
>>> > >>>> > >> > associated with a down node. Let hold on this topic for
>>> future
>>> > >>>> > >> discussion.
>>> > >>>> > >> >
>>> > >>>> > >> > Going by the explanation where the master/slave nodes have
>>> > >>>> certain
>>> > >>>> > >> > predefined tasks to perform is perfectly fine.
>>> > >>>> > >> >
>>> > >>>> > >> > I have this scenario,
>>> > >>>> > >> >
>>> > >>>> > >> > I am using HAProxy as my load balancer and configured 3
>>> nodes
>>> > in
>>> > >>>> the
>>> > >>>> > >> > cluster.
>>> > >>>> > >> >
>>> > >>>> > >> > Node1 - Active
>>> > >>>> > >> > Node2 - Active
>>> > >>>> > >> > Node3 - Backup
>>> > >>>> > >> >
>>> > >>>> > >> > Load balancing algorithm: RoundRobin
>>> > >>>> > >> >
>>> > >>>> > >> > A Backup node (Node3) is one which the load balancer will
>>> not
>>> > >>>> route
>>> > >>>> > >> > requests to, until one of the Active node i.e either Node1
>>> or
>>> > >>>> Node2
>>> > >>>> > has
>>> > >>>> > >> > gone down.
>>> > >>>> > >> >
>>> > >>>> > >> > All these 3 nodes are also part of the hazelcast cluster as
>>> > well.
>>> > >>>> > >> >
>>> > >>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
>>> > >>>> leader/master
>>> > >>>> > >> and
>>> > >>>> > >> > Node2,Node3 as slaves.
>>> > >>>> > >> >
>>> > >>>> > >> > I initiate the deploy operation on the DeploymentWebService
>>> > >>>> which the
>>> > >>>> > >> load
>>> > >>>> > >> > balancer routes it to one of the Active nodes in the
>>> cluster,
>>> > >>>> lets say
>>> > >>>> > >> it's
>>> > >>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
>>> > >>>> cluster,
>>> > >>>> > >> > deployment is a success.
>>> > >>>> > >> >
>>> > >>>> > >> > I initiate another deploy operation on the
>>> DeploymentWebService
>>> > >>>> which
>>> > >>>> > >> the
>>> > >>>> > >> > load balancer routes it to the next active node which is
>>> Node2.
>>> > >>>> Since
>>> > >>>> > >> Node2
>>> > >>>> > >> > is a slave in the Hazelcast cluster, What happens to the
>>> > >>>> deployment?
>>> > >>>> > >> >
>>> > >>>> > >> > regards,
>>> > >>>> > >> > sathwik
>>> > >>>> > >> >
>>> > >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>>> > >>>> > >> > suba.11@cse.mrt.ac.lk
>>> > >>>> > >> > > wrote:
>>> > >>>> > >> >
>>> > >>>> > >> > > Hi,
>>> > >>>> > >> > >
>>> > >>>> > >> > > I will explain my approach as much as possible. The
>>> oldest
>>> > >>>> node in
>>> > >>>> > the
>>> > >>>> > >> > > hazelcast cluster is elected as the master node. In the
>>> > >>>> failure of
>>> > >>>> > the
>>> > >>>> > >> > > master node, next oldest node will be elected as the
>>> master
>>> > >>>> node.
>>> > >>>> > This
>>> > >>>> > >> > > master-slave configuration is just for deployment. When
>>> the
>>> > >>>> > hazelcast
>>> > >>>> > >> > > cluster elected the master node, that node becomes a
>>> master
>>> > >>>> node for
>>> > >>>> > >> > > deploying process. So it will do the deploying
>>> artifacts. If
>>> > >>>> you
>>> > >>>> > want
>>> > >>>> > >> to
>>> > >>>> > >> > > get the idea of electing master node please refer the
>>> code
>>> > >>>> which I
>>> > >>>> > >> have
>>> > >>>> > >> > > located in the github. (
>>> > >>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>>> > >>>> > >> > >
>>> > >>>> > >> > > I identified separated actions which should be followed
>>> by
>>> > the
>>> > >>>> > master
>>> > >>>> > >> and
>>> > >>>> > >> > > salve nodes.
>>> > >>>> > >> > > Actions which are followed by master node only
>>> > >>>> > >> > > 1) create deployment unit
>>> > >>>> > >> > > 2) set the version nu to deployment unit
>>> > >>>> > >> > > 3) compile deployment unit
>>> > >>>> > >> > > 4) scan deployment unit
>>> > >>>> > >> > > 5) retire previous versions
>>> > >>>> > >> > > Master node and slave nodes should create _processes
>>> which
>>> > >>>> stores
>>> > >>>> > >> > > ProcessConfImpl
>>> > >>>> > >> > > Only master node will write the version nu to database,
>>> > create
>>> > >>>> > >> .deployed
>>> > >>>> > >> > > file
>>> > >>>> > >> > >
>>> > >>>> > >> > > So there are some actions which should be followed only
>>> by
>>> > >>>> master
>>> > >>>> > node
>>> > >>>> > >> > > while other actions should be followed by all the
>>> nodes.The
>>> > >>>> idea of
>>> > >>>> > >> > having
>>> > >>>> > >> > > a master node is deploying artifacts and avoid others
>>> from
>>> > >>>> writing
>>> > >>>> > the
>>> > >>>> > >> > > version nu to database.
>>> > >>>> > >> > > Whether a node is active or passive, all nodes should do
>>> the
>>> > >>>> > >> > > deployment.Master
>>> > >>>> > >> > > and slaves will follow necessary actions as in above.
>>> > >>>> > >> > >
>>> > >>>> > >> > >
>>> > >>>> > >> > >
>>> > >>>> > >> > >
>>> > >>>> > >> > >
>>> > >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <
>>> sathwik.bp@gmail.com>
>>> > >>>> wrote:
>>> > >>>> > >> > >
>>> > >>>> > >> > > > Nandika,
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > I very well understand what you have put across, but
>>> it's
>>> > >>>> > secondary
>>> > >>>> > >> to
>>> > >>>> > >> > me
>>> > >>>> > >> > > > now.
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > Sudharma,
>>> > >>>> > >> > > > My primary concern is to understand at a high level the
>>> > >>>> deployment
>>> > >>>> > >> > > > architecture and how would master-slave configuration
>>> fit
>>> > >>>> in. Are
>>> > >>>> > >> there
>>> > >>>> > >> > > any
>>> > >>>> > >> > > > restrictions imposed by the in-progress design?
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > Firstly, how would ODE process deployment work under
>>> these
>>> > >>>> cluster
>>> > >>>> > >> > > > configurations?
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > Sample Cluster configurations: A load balancer is
>>> > >>>> frontending the
>>> > >>>> > >> > > servers.
>>> > >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>>> > >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>>> > >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either
>>> in
>>> > >>>> Active or
>>> > >>>> > >> > > Passive.
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > regards,
>>> > >>>> > >> > > > sathwik
>>> > >>>> > >> > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>>> > >>>> > >> > jayawark@gmail.com
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > wrote:
>>> > >>>> > >> > > >
>>> > >>>> > >> > > > > Hi Sathwik,
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > > According to my understanding, in the clustering
>>> > scenario,
>>> > >>>> the
>>> > >>>> > >> master
>>> > >>>> > >> > > > node
>>> > >>>> > >> > > > > should perform all the deployment actions and the
>>> slave
>>> > >>>> nodes
>>> > >>>> > also
>>> > >>>> > >> > need
>>> > >>>> > >> > > > to
>>> > >>>> > >> > > > > perform some deployment actions. For example, the
>>> slave
>>> > >>>> nodes
>>> > >>>> > also
>>> > >>>> > >> > > should
>>> > >>>> > >> > > > > handle the process ACTIVATED event so that the
>>> process
>>> > >>>> > >> configuration
>>> > >>>> > >> > is
>>> > >>>> > >> > > > > added to the engine and necessary web services are
>>> > created
>>> > >>>> so
>>> > >>>> > that
>>> > >>>> > >> > when
>>> > >>>> > >> > > > the
>>> > >>>> > >> > > > > load balancer send requests to any node in the
>>> cluster,
>>> > it
>>> > >>>> is
>>> > >>>> > >> ready
>>> > >>>> > >> > to
>>> > >>>> > >> > > > > accept those requests.
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > > Regards
>>> > >>>> > >> > > > > Nandika
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>>> > >>>> > >> sathwik.bp@gmail.com>
>>> > >>>> > >> > > > > wrote:
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > > > > Sudharma,
>>> > >>>> > >> > > > > >
>>> > >>>> > >> > > > > > Where are you going to configure the
>>> master-slaves, is
>>> > >>>> it in
>>> > >>>> > the
>>> > >>>> > >> > web
>>> > >>>> > >> > > > > > application level or at the load balancer?
>>> > >>>> > >> > > > > >
>>> > >>>> > >> > > > > > regards,
>>> > >>>> > >> > > > > > sathwik
>>> > >>>> > >> > > > > >
>>> > >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma
>>> subasinghe <
>>> > >>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
>>> > >>>> > >> > > > > > wrote:
>>> > >>>> > >> > > > > >
>>> > >>>> > >> > > > > > > Hi Tammo,
>>> > >>>> > >> > > > > > >
>>> > >>>> > >> > > > > > > Can you suggest the best method from these to
>>> > >>>> implement? As
>>> > >>>> > >> > first I
>>> > >>>> > >> > > > > > > suggested the master-slaves scenario I think it
>>> is
>>> > >>>> easy to
>>> > >>>> > >> > > implement
>>> > >>>> > >> > > > > than
>>> > >>>> > >> > > > > > > distributed lock scenario. However if you can
>>> suggest
>>> > >>>> one
>>> > >>>> > from
>>> > >>>> > >> > > these
>>> > >>>> > >> > > > > two,
>>> > >>>> > >> > > > > > > then I can think about it.
>>> > >>>> > >> > > > > > >
>>> > >>>> > >> > > > > > > Thank you
>>> > >>>> > >> > > > > > >
>>> > >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
>>> > >>>> sathwik.bp@gmail.com>
>>> > >>>> > >> > wrote:
>>> > >>>> > >> > > > > > >
>>> > >>>> > >> > > > > > > > With respect to the hotdeployment,
>>> > >>>> > >> > > > > > > >
>>> > >>>> > >> > > > > > > > We can drop the deployment archive onto the
>>> > >>>> deployment
>>> > >>>> > >> folder.
>>> > >>>> > >> > > > Since
>>> > >>>> > >> > > > > > the
>>> > >>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed
>>> lock
>>> > >>>> for
>>> > >>>> > the
>>> > >>>> > >> > > > > > > DeploymentUnit,
>>> > >>>> > >> > > > > > > > only one of the nodes will get the lock and
>>> > initiate
>>> > >>>> the
>>> > >>>> > >> > > > deployment.
>>> > >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
>>> > >>>> acquiring
>>> > >>>> > the
>>> > >>>> > >> > lock
>>> > >>>> > >> > > > and
>>> > >>>> > >> > > > > > > hence
>>> > >>>> > >> > > > > > > > will silently ignore it.
>>> > >>>> > >> > > > > > > >
>>> > >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>>> > >>>> > >> > > > sathwik.bp@gmail.com>
>>> > >>>> > >> > > > > > > > wrote:
>>> > >>>> > >> > > > > > > >
>>> > >>>> > >> > > > > > > > > Hi Tammo,
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > The distributed lock acquisition on the
>>> > >>>> DeploymentUnit
>>> > >>>> > >> should
>>> > >>>> > >> > > be
>>> > >>>> > >> > > > > > added
>>> > >>>> > >> > > > > > > to
>>> > >>>> > >> > > > > > > > > both DeploymentWebService and
>>> DeploymentPoller.
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > When a deployment operation is initiated
>>> through
>>> > >>>> the
>>> > >>>> > >> > > > > > > > DeploymentWebService,
>>> > >>>> > >> > > > > > > > > The load balancer routes it to any of the
>>> > available
>>> > >>>> > nodes.
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
>>> > >>>> acquires
>>> > >>>> > the
>>> > >>>> > >> > > > > Distributed
>>> > >>>> > >> > > > > > > > > lock. On the remaining nodes the
>>> DeploymentPoller
>>> > >>>> will
>>> > >>>> > >> try to
>>> > >>>> > >> > > > > acquire
>>> > >>>> > >> > > > > > > the
>>> > >>>> > >> > > > > > > > > distributed lock and will not get it and
>>> hence
>>> > will
>>> > >>>> > >> silently
>>> > >>>> > >> > > > ignore
>>> > >>>> > >> > > > > > it.
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > Once the routed node completes the
>>> deployment, it
>>> > >>>> will
>>> > >>>> > >> > release
>>> > >>>> > >> > > > the
>>> > >>>> > >> > > > > > > lock.
>>> > >>>> > >> > > > > > > > > This way we don't have to stall the
>>> > >>>> DeploymentPoller in
>>> > >>>> > >> other
>>> > >>>> > >> > > > > nodes.
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > Does it answer the concerns?
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > Now, if we give the responsibility of
>>> identifying
>>> > >>>> the
>>> > >>>> > >> master
>>> > >>>> > >> > > node
>>> > >>>> > >> > > > > to
>>> > >>>> > >> > > > > > > the
>>> > >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the
>>> load
>>> > >>>> balancer
>>> > >>>> > to
>>> > >>>> > >> > > change
>>> > >>>> > >> > > > > > it's
>>> > >>>> > >> > > > > > > > > configuration about the master node?
>>> > >>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
>>> > >>>> > >> > > > > > > > > node1 -master
>>> > >>>> > >> > > > > > > > > node2 - slave
>>> > >>>> > >> > > > > > > > > node3 - slave
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as
>>> > >>>> master
>>> > >>>> > node,
>>> > >>>> > >> > but
>>> > >>>> > >> > > > > > > hazelcast
>>> > >>>> > >> > > > > > > > > might promote Node3 as master node. They are
>>> out
>>> > of
>>> > >>>> > sync.
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > Is this argument valid?
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > regards,
>>> > >>>> > >> > > > > > > > > sathwik
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van
>>> > Lessen <
>>> > >>>> > >> > > > > > > tvanlessen@gmail.com>
>>> > >>>> > >> > > > > > > > > wrote:
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > >> Hi Sudharma,
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >> what do you expect from the "other nodes
>>> > >>>> deployment"?
>>> > >>>> > >> > > > Compilation
>>> > >>>> > >> > > > > is
>>> > >>>> > >> > > > > > > not
>>> > >>>> > >> > > > > > > > >> needed since the CBP file is written to the
>>> > >>>> (shared)
>>> > >>>> > FS.
>>> > >>>> > >> > > > > > Registration
>>> > >>>> > >> > > > > > > is
>>> > >>>> > >> > > > > > > > >> also not needed, since it is done via the
>>> shared
>>> > >>>> > >> database.
>>> > >>>> > >> > So
>>> > >>>> > >> > > > the
>>> > >>>> > >> > > > > > only
>>> > >>>> > >> > > > > > > > >> thing that might be needed is to tell the
>>> engine
>>> > >>>> that
>>> > >>>> > >> there
>>> > >>>> > >> > > is a
>>> > >>>> > >> > > > > new
>>> > >>>> > >> > > > > > > > >> deployment. I'd need to check that. If this
>>> is
>>> > >>>> needed,
>>> > >>>> > I
>>> > >>>> > >> > > revert
>>> > >>>> > >> > > > my
>>> > >>>> > >> > > > > > > last
>>> > >>>> > >> > > > > > > > >> statement, then it is perhaps better to just
>>> > send
>>> > >>>> an
>>> > >>>> > >> event
>>> > >>>> > >> > > over
>>> > >>>> > >> > > > > > > > Hazelcast
>>> > >>>> > >> > > > > > > > >> to all nodes that the deployment has
>>> changed.
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >> Best,
>>> > >>>> > >> > > > > > > > >>   Tammo
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
>>> > >>>> subasinghe <
>>> > >>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
>>> > >>>> > >> > > > > > > > >> > wrote:
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >> > Hi Tammo,
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >> > The master node writes meta data. But
>>> runtime
>>> > >>>> > >> information
>>> > >>>> > >> > > must
>>> > >>>> > >> > > > > be
>>> > >>>> > >> > > > > > > > >> available
>>> > >>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared,
>>> all
>>> > >>>> nodes
>>> > >>>> > will
>>> > >>>> > >> > see
>>> > >>>> > >> > > > the
>>> > >>>> > >> > > > > > > > >> > availability of a new process. My idea is
>>> for
>>> > >>>> master
>>> > >>>> > >> node
>>> > >>>> > >> > to
>>> > >>>> > >> > > > > write
>>> > >>>> > >> > > > > > > the
>>> > >>>> > >> > > > > > > > >> meta
>>> > >>>> > >> > > > > > > > >> > data and other nodes to just read the meta
>>> > data
>>> > >>>> and
>>> > >>>> > >> load
>>> > >>>> > >> > > > > > process.So
>>> > >>>> > >> > > > > > > we
>>> > >>>> > >> > > > > > > > >> need
>>> > >>>> > >> > > > > > > > >> > a small delay between master node
>>> deployment
>>> > and
>>> > >>>> > other
>>> > >>>> > >> > nodes
>>> > >>>> > >> > > > > > > > deployment.
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >> > Is there anyway to set the delay between
>>> > master
>>> > >>>> node
>>> > >>>> > >> and
>>> > >>>> > >> > > > slaves
>>> > >>>> > >> > > > > > > until
>>> > >>>> > >> > > > > > > > >> > master node finish the deployment?
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >> > Thank you
>>> > >>>> > >> > > > > > > > >> > Sudharma
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen
>>> <
>>> > >>>> > >> > > > tvanlessen@gmail.com
>>> > >>>> > >> > > > > >
>>> > >>>> > >> > > > > > > > wrote:
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >> > > Hi Sathwik,
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM,
>>> Sathwik B
>>> > P <
>>> > >>>> > >> > > > > > > sathwik.bp@gmail.com>
>>> > >>>> > >> > > > > > > > >> > wrote:
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
>>> > >>>> > >> > > > > > > > >> > > >
>>> > >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is
>>> the
>>> > >>>> master
>>> > >>>> > >> node
>>> > >>>> > >> > in
>>> > >>>> > >> > > > the
>>> > >>>> > >> > > > > > > > cluster?
>>> > >>>> > >> > > > > > > > >> > > >
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > I think the easiest approach is to
>>> always
>>> > >>>> elect the
>>> > >>>> > >> > oldest
>>> > >>>> > >> > > > > node
>>> > >>>> > >> > > > > > in
>>> > >>>> > >> > > > > > > > the
>>> > >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK
>>> Hazelcast
>>> > can
>>> > >>>> > easily
>>> > >>>> > >> > asked
>>> > >>>> > >> > > > for
>>> > >>>> > >> > > > > > > this
>>> > >>>> > >> > > > > > > > >> > > information.
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the
>>> Deployment
>>> > >>>> Pollers
>>> > >>>> > in
>>> > >>>> > >> > the
>>> > >>>> > >> > > > > slave
>>> > >>>> > >> > > > > > > > nodes?
>>> > >>>> > >> > > > > > > > >> > > >
>>> > >>>> > >> > > > > > > > >> > > >
>>> > >>>> > >> > > > > > > > >> > > Absolutely.
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > Suggestion:
>>> > >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
>>> > >>>> Master-SLaves.
>>> > >>>> > Why
>>> > >>>> > >> > not
>>> > >>>> > >> > > > give
>>> > >>>> > >> > > > > > > every
>>> > >>>> > >> > > > > > > > >> node
>>> > >>>> > >> > > > > > > > >> > > in
>>> > >>>> > >> > > > > > > > >> > > > the cluster the same status
>>> > (Active-Active).
>>> > >>>> > >> > > > > > > > >> > > >
>>> > >>>> > >> > > > > > > > >> > > > When a new deployment is made, the
>>> load
>>> > >>>> balancer
>>> > >>>> > >> can
>>> > >>>> > >> > > push
>>> > >>>> > >> > > > it
>>> > >>>> > >> > > > > > to
>>> > >>>> > >> > > > > > > > any
>>> > >>>> > >> > > > > > > > >> of
>>> > >>>> > >> > > > > > > > >> > > the
>>> > >>>> > >> > > > > > > > >> > > > available nodes. That node will
>>> probably
>>> > >>>> acquire
>>> > >>>> > a
>>> > >>>> > >> > > > > distributed
>>> > >>>> > >> > > > > > > > lock
>>> > >>>> > >> > > > > > > > >> on
>>> > >>>> > >> > > > > > > > >> > > the
>>> > >>>> > >> > > > > > > > >> > > > deployment unit and acts as master for
>>> > that
>>> > >>>> > >> > deployment.
>>> > >>>> > >> > > > This
>>> > >>>> > >> > > > > > > > ensures
>>> > >>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
>>> > >>>> Probably no
>>> > >>>> > >> static
>>> > >>>> > >> > > > > > > > >> configuration of
>>> > >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor
>>> in
>>> > the
>>> > >>>> > >> > hazelcast.
>>> > >>>> > >> > > > > > > > >> > > >
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > But this would not allow to have the
>>> > >>>> hotdeployment
>>> > >>>> > >> via
>>> > >>>> > >> > > > > > filesystem
>>> > >>>> > >> > > > > > > > >> still
>>> > >>>> > >> > > > > > > > >> > > enabled, right?
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > Best,
>>> > >>>> > >> > > > > > > > >> > >   Tammo
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> > > --
>>> > >>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>>> > >>>> > >> > > > > > > > >> > >
>>> > >>>> > >> > > > > > > > >> >
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >> --
>>> > >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>>> > >>>> > >> > > > > > > > >>
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > > >
>>> > >>>> > >> > > > > > > >
>>> > >>>> > >> > > > > > >
>>> > >>>> > >> > > > > >
>>> > >>>> > >> > > > >
>>> > >>>> > >> > > >
>>> > >>>> > >> > >
>>> > >>>> > >> >
>>> > >>>> > >>
>>> > >>>> > >
>>> > >>>> > >
>>> > >>>> >
>>> > >>>>
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Sathwik,

I tried to implement your logic also. Following is the link for committed
code upto now.
https://github.com/Subasinghe/ode/tree/ODECluster

I used info msgs to debugging as it can be observed easily in console. When
releasing the lock acquired by poller or web service the state is set to
false as in my code. But sometimes it is set to true and at that time
second server's value is also true.  Following is the related log for each
server.

Server 1
02:12:02,957 INFO  [DeploymentPoller] Trying to access the lock for
MagicSession
02:12:02,962 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value for
MagicSession file after locking: true
02:12:03,838 INFO  [BpelServerImpl] Registered process {
http://ode/bpel/unit-test}MagicSessionMain-3.
02:12:04,015 INFO  [BpelServerImpl] Registered process {
http://ode/bpel/responder}MagicSessionResponder-3.
02:12:04,015 INFO  [DeploymentPoller] Deployment of artifact MagicSession
successful: [{http://ode/bpel/unit-test}MagicSessionMain-3, {
http://ode/bpel/responder}MagicSessionResponder-3]
02:12:04,015 INFO  [DeploymentPoller] Trying to release the lock for
MagicSession
02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:63 duLocked value for
MagicSession file after unlocking: true

Server 2
02:12:03,119 INFO  [DeploymentPoller] Trying to access the lock for
MagicSession
02:12:04,017 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value for
MagicSession file after locking: true
02:12:04,017 INFO  [DeploymentPoller] Trying to release the lock for
MagicSession
02:12:04,019 INFO  [HazelcastClusterImpl] ThreadID:61 duLocked value for
MagicSession file after unlocking: false

Is this possible? But there were not any conflicts while deploying. It
worked perfectly.

Thank you


On 7 June 2015 at 11:10, sudharma subasinghe <su...@cse.mrt.ac.lk> wrote:

> Hi Sathwik,
>
> I will concern your approach. Thank you for your effort.
>
> On 6 June 2015 at 18:43, Sathwik B P <sa...@gmail.com> wrote:
>
>> I have attached to the JIRA https://issues.apache.org/jira/browse/ODE-563
>>
>> On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <
>> suba.11@cse.mrt.ac.lk>
>> wrote:
>>
>> > Hi Sathwik,
>> >
>> > I can't find the attached document. Please kind be enough to resend it.
>> >
>> > On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:
>> >
>> > > Refer this one as I have corrected the numbering of steps.
>> > >
>> > > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com>
>> > wrote:
>> > >
>> > >> Hi Sudharma/Tammo,
>> > >>
>> > >> Kindly review attached document on my proposed approach. Let me know
>> if
>> > >> you have any concerns or doubts on it.
>> > >>
>> > >> regards,
>> > >> sathwik
>> > >>
>> > >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com>
>> > wrote:
>> > >>
>> > >>> find response inline
>> > >>>
>> > >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
>> > >>> suba.11@cse.mrt.ac.lk> wrote:
>> > >>>
>> > >>>> Hi,
>> > >>>>
>> > >>>> Sorry for the late reply. I was trying to achieve a proper
>> solution.
>> > >>>> Following is my approach.
>> > >>>>
>> > >>>> 1) I elected master node for deploying purpose. So when a master
>> node
>> > >>>> goes
>> > >>>> down hazelcast will elect the next oldest node for master.
>> > >>>>
>> > >>>
>> > >>>     Perfect
>> > >>>
>> > >>>
>> > >>>> 2) ODEServer can identify whether clustering is enabled or not by
>> > >>>> getting
>> > >>>> the property value in ode-axis2.properties file. So I introduced
>> new
>> > >>>> property called "ode-axis2.hazelcast.clustering.enabled". If there
>> is
>> > no
>> > >>>> clustering enabled server will work as it is. If clustering is
>> > enabled,
>> > >>>> cluster will be initialized.
>> > >>>>
>> > >>>
>> > >>>     Perfect
>> > >>>
>> > >>>
>> > >>>> 3) In manual deployment its responsibility is taken by the
>> > >>>> DeploymentPoller. So I give the deployment capability to master
>> node,s
>> > >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to
>> true.So
>> > >>>> others will not be able to go into check() method in
>> DeploymentPoller.
>> > >>>> So
>> > >>>> deployment will be done by only the master node.
>> > >>>>
>> > >>>
>> > >>>     Perfect
>> > >>>
>> > >>>
>> > >>>>
>> > >>>> 4) In DeploymentWebService, I had to consider few cases.If the
>> deploy
>> > >>>> request goes to the master node, it will deploy the process through
>> > web
>> > >>>> service.Others pollers will not go into check() method as they are
>> not
>> > >>>> masters. So master can continue without any involvement of others.
>> > >>>>
>> > >>>>
>> > >>>     Perfect
>> > >>>
>> > >>>
>> > >>>> 5) If the deploy request goes to a slave node, it will do up to
>> file
>> > >>>> creation in the file system.Slave will be stopped at that point. As
>> > only
>> > >>>> master poller is checking, master can continue from created files
>> in
>> > the
>> > >>>> file system.
>> > >>>>
>> > >>>
>> > >>>     DeploymentWebService provides synchronous operations. The
>> status of
>> > >>> the operation should be communicated to the calling client in the
>> same
>> > call.
>> > >>>     DeploymentPoller is a backend thread that goes over each and
>> every
>> > >>> directory under the Deployment directory checking for any changes to
>> > >>> deploy.xml in existing processes and deploy newly added processes.
>> This
>> > >>> process is  sequential and time consuming. As the process
>> directories
>> > >>> grows, so does the time taken for execution of the thread.
>> > >>>
>> > >>>     Since the request is on a slave node and the processing is done
>> on
>> > >>> master node, how do you check for the completion of the
>> > >>> deployment/undeployment of processes and respond back to the client
>> > since
>> > >>> the web service call is a synchronous operation. As
>> DeploymentPoller is
>> > >>> taking a lot of time in processing, your request will time out
>> right.
>> > >>>
>> > >>>
>> > >>>>
>> > >>>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl.
>> > Each
>> > >>>> _deploymentUnits stores only what its server has deployed. So
>> think,
>> > >>>> that a
>> > >>>> master node goes down another master node appears.But its
>> > >>>> __deploymentUnits
>> > >>>> does not have dus which has deployed by the earlier master node.
>> Hence
>> > >>>> it
>> > >>>> will not be able retire earlier version of the process which is
>> > >>>> deployed by
>> > >>>> previous master. So there will two process which are in "ACTIVE"
>> state
>> > >>>
>> > >>>
>> > >>>> 7) To avoid this, I add the ODEServer as an Observer to check when
>> a
>> > new
>> > >>>> master is electing, then load all the deployment units out of the
>> > >>>> store. So
>> > >>>> new master node can have all the dus and can retire appropriate
>> > version.
>> > >>>> Usually loadAll() is called at the server start-up time. But there
>> is
>> > no
>> > >>>> other way to solve this. I tried to use Hazelcast IMap to store all
>> > dus
>> > >>>> among all nodes. But it wasn't success as du is not serializable
>> > >>>> object.
>> > >>>>
>> > >>>
>> > >>>> 8) I figured out that we do not need send cluster message to
>> others as
>> > >>>> all
>> > >>>> the dus' data are persisted to the shared DB. So each node can take
>> > the
>> > >>>> du
>> > >>>> and retrieve necessary data using already implemented methods in
>> > Process
>> > >>>> Store.
>> > >>>>
>> > >>>> 9) But there is an another problem.The axis2 service corresponding
>> to
>> > a
>> > >>>> deployed process does not appear on all nodes of the cluster. That
>> is
>> > >>>> because each server add du which is deployed by it to the process
>> > >>>> store.That is why I had use loadAll() when masters are changing.
>> How
>> > to
>> > >>>> solve this?
>> > >>>>
>> > >>>
>> > >>>     I do appriciate your efforts in understanding the implmentation
>> and
>> > >>> changes that need to be done. You are bang on it.
>> > >>>     fireEvent(..) is the method that triggers process activation and
>> > >>> necessary service creation.
>> > >>>
>> > >>>
>> > >>>     But with this given apporach from steps 6 to 8, ODE cannot have
>> > >>> atleast 2 Active servers for Load balancing. You are concentrating
>> on
>> > only
>> > >>> one active node that will do deployments and cater to process
>> > invocations.
>> > >>>     We should also think about scaling ODE to multiple servers to
>> > handle
>> > >>> load.
>> > >>>
>> > >>>     What do you think.
>> > >>>
>> > >>>
>> > >>>> Thank you,
>> > >>>> Sudharma
>> > >>>>
>> > >>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
>> > >>>>
>> > >>>> > Sudharma,
>> > >>>> >
>> > >>>> > Any updates?
>> > >>>> >
>> > >>>> > regards,
>> > >>>> > sathwik
>> > >>>> >
>> > >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <
>> sathwik.bp@gmail.com>
>> > >>>> wrote:
>> > >>>> >
>> > >>>> > > Sudharma,
>> > >>>> > >
>> > >>>> > > Can you elaborate on your option 1).
>> > >>>> > >
>> > >>>> > > Response to your option 2).
>> > >>>> > >
>> > >>>> > >     Process Store is the component that handles process
>> metadata,
>> > >>>> > > compilation and deployment in ODE. Integration layers in ODE
>> > >>>> (Axis2, JBI)
>> > >>>> > > use the process store.
>> > >>>> > >     Future implementations of IL for ODE will also use the
>> process
>> > >>>> store.
>> > >>>> > > We should not be thinking of moving the process store
>> > functionality
>> > >>>> to
>> > >>>> > the
>> > >>>> > > integration layers.
>> > >>>> > >
>> > >>>> > >
>> > >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
>> > >>>> > > suba.11@cse.mrt.ac.lk> wrote:
>> > >>>> > >
>> > >>>> > >> Hi,
>> > >>>> > >>
>> > >>>> > >> I understood the problem within dynamic master/slave
>> > >>>> configuration. In
>> > >>>> > my
>> > >>>> > >> approach, when a deployment request is routed to a slave node
>> > >>>> there will
>> > >>>> > >> not be a deployment. I suggest two options to avoid it.
>> > >>>> > >> 1) Have static master/slave configuration only for deploy
>> process
>> > >>>> > >>
>> > >>>> > > 2) Modify the deployment web service to complie and verify the
>> > >>>> process
>> > >>>> > and
>> > >>>> > >> then copy it to the deploy folder irrespective of whether its
>> a
>> > >>>> master
>> > >>>> > or
>> > >>>> > >> slave, then deployment poller should take care of the
>> deployment
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >
>> > >>>> > >>
>> > >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com>
>> > wrote:
>> > >>>> > >>
>> > >>>> > >> > Sudharma,
>> > >>>> > >> >
>> > >>>> > >> > We definitely need a master/slave in the hazelcast cluster.
>> > This
>> > >>>> is
>> > >>>> > >> > probably needed for the job migration in the Scheduler to
>> > >>>> migrate the
>> > >>>> > >> jobs
>> > >>>> > >> > associated with a down node. Let hold on this topic for
>> future
>> > >>>> > >> discussion.
>> > >>>> > >> >
>> > >>>> > >> > Going by the explanation where the master/slave nodes have
>> > >>>> certain
>> > >>>> > >> > predefined tasks to perform is perfectly fine.
>> > >>>> > >> >
>> > >>>> > >> > I have this scenario,
>> > >>>> > >> >
>> > >>>> > >> > I am using HAProxy as my load balancer and configured 3
>> nodes
>> > in
>> > >>>> the
>> > >>>> > >> > cluster.
>> > >>>> > >> >
>> > >>>> > >> > Node1 - Active
>> > >>>> > >> > Node2 - Active
>> > >>>> > >> > Node3 - Backup
>> > >>>> > >> >
>> > >>>> > >> > Load balancing algorithm: RoundRobin
>> > >>>> > >> >
>> > >>>> > >> > A Backup node (Node3) is one which the load balancer will
>> not
>> > >>>> route
>> > >>>> > >> > requests to, until one of the Active node i.e either Node1
>> or
>> > >>>> Node2
>> > >>>> > has
>> > >>>> > >> > gone down.
>> > >>>> > >> >
>> > >>>> > >> > All these 3 nodes are also part of the hazelcast cluster as
>> > well.
>> > >>>> > >> >
>> > >>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
>> > >>>> leader/master
>> > >>>> > >> and
>> > >>>> > >> > Node2,Node3 as slaves.
>> > >>>> > >> >
>> > >>>> > >> > I initiate the deploy operation on the DeploymentWebService
>> > >>>> which the
>> > >>>> > >> load
>> > >>>> > >> > balancer routes it to one of the Active nodes in the
>> cluster,
>> > >>>> lets say
>> > >>>> > >> it's
>> > >>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
>> > >>>> cluster,
>> > >>>> > >> > deployment is a success.
>> > >>>> > >> >
>> > >>>> > >> > I initiate another deploy operation on the
>> DeploymentWebService
>> > >>>> which
>> > >>>> > >> the
>> > >>>> > >> > load balancer routes it to the next active node which is
>> Node2.
>> > >>>> Since
>> > >>>> > >> Node2
>> > >>>> > >> > is a slave in the Hazelcast cluster, What happens to the
>> > >>>> deployment?
>> > >>>> > >> >
>> > >>>> > >> > regards,
>> > >>>> > >> > sathwik
>> > >>>> > >> >
>> > >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>> > >>>> > >> > suba.11@cse.mrt.ac.lk
>> > >>>> > >> > > wrote:
>> > >>>> > >> >
>> > >>>> > >> > > Hi,
>> > >>>> > >> > >
>> > >>>> > >> > > I will explain my approach as much as possible. The oldest
>> > >>>> node in
>> > >>>> > the
>> > >>>> > >> > > hazelcast cluster is elected as the master node. In the
>> > >>>> failure of
>> > >>>> > the
>> > >>>> > >> > > master node, next oldest node will be elected as the
>> master
>> > >>>> node.
>> > >>>> > This
>> > >>>> > >> > > master-slave configuration is just for deployment. When
>> the
>> > >>>> > hazelcast
>> > >>>> > >> > > cluster elected the master node, that node becomes a
>> master
>> > >>>> node for
>> > >>>> > >> > > deploying process. So it will do the deploying artifacts.
>> If
>> > >>>> you
>> > >>>> > want
>> > >>>> > >> to
>> > >>>> > >> > > get the idea of electing master node please refer the code
>> > >>>> which I
>> > >>>> > >> have
>> > >>>> > >> > > located in the github. (
>> > >>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>> > >>>> > >> > >
>> > >>>> > >> > > I identified separated actions which should be followed by
>> > the
>> > >>>> > master
>> > >>>> > >> and
>> > >>>> > >> > > salve nodes.
>> > >>>> > >> > > Actions which are followed by master node only
>> > >>>> > >> > > 1) create deployment unit
>> > >>>> > >> > > 2) set the version nu to deployment unit
>> > >>>> > >> > > 3) compile deployment unit
>> > >>>> > >> > > 4) scan deployment unit
>> > >>>> > >> > > 5) retire previous versions
>> > >>>> > >> > > Master node and slave nodes should create _processes which
>> > >>>> stores
>> > >>>> > >> > > ProcessConfImpl
>> > >>>> > >> > > Only master node will write the version nu to database,
>> > create
>> > >>>> > >> .deployed
>> > >>>> > >> > > file
>> > >>>> > >> > >
>> > >>>> > >> > > So there are some actions which should be followed only by
>> > >>>> master
>> > >>>> > node
>> > >>>> > >> > > while other actions should be followed by all the
>> nodes.The
>> > >>>> idea of
>> > >>>> > >> > having
>> > >>>> > >> > > a master node is deploying artifacts and avoid others from
>> > >>>> writing
>> > >>>> > the
>> > >>>> > >> > > version nu to database.
>> > >>>> > >> > > Whether a node is active or passive, all nodes should do
>> the
>> > >>>> > >> > > deployment.Master
>> > >>>> > >> > > and slaves will follow necessary actions as in above.
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <
>> sathwik.bp@gmail.com>
>> > >>>> wrote:
>> > >>>> > >> > >
>> > >>>> > >> > > > Nandika,
>> > >>>> > >> > > >
>> > >>>> > >> > > > I very well understand what you have put across, but
>> it's
>> > >>>> > secondary
>> > >>>> > >> to
>> > >>>> > >> > me
>> > >>>> > >> > > > now.
>> > >>>> > >> > > >
>> > >>>> > >> > > > Sudharma,
>> > >>>> > >> > > > My primary concern is to understand at a high level the
>> > >>>> deployment
>> > >>>> > >> > > > architecture and how would master-slave configuration
>> fit
>> > >>>> in. Are
>> > >>>> > >> there
>> > >>>> > >> > > any
>> > >>>> > >> > > > restrictions imposed by the in-progress design?
>> > >>>> > >> > > >
>> > >>>> > >> > > > Firstly, how would ODE process deployment work under
>> these
>> > >>>> cluster
>> > >>>> > >> > > > configurations?
>> > >>>> > >> > > >
>> > >>>> > >> > > > Sample Cluster configurations: A load balancer is
>> > >>>> frontending the
>> > >>>> > >> > > servers.
>> > >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>> > >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>> > >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
>> > >>>> Active or
>> > >>>> > >> > > Passive.
>> > >>>> > >> > > >
>> > >>>> > >> > > > regards,
>> > >>>> > >> > > > sathwik
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>> > >>>> > >> > jayawark@gmail.com
>> > >>>> > >> > > >
>> > >>>> > >> > > > wrote:
>> > >>>> > >> > > >
>> > >>>> > >> > > > > Hi Sathwik,
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > According to my understanding, in the clustering
>> > scenario,
>> > >>>> the
>> > >>>> > >> master
>> > >>>> > >> > > > node
>> > >>>> > >> > > > > should perform all the deployment actions and the
>> slave
>> > >>>> nodes
>> > >>>> > also
>> > >>>> > >> > need
>> > >>>> > >> > > > to
>> > >>>> > >> > > > > perform some deployment actions. For example, the
>> slave
>> > >>>> nodes
>> > >>>> > also
>> > >>>> > >> > > should
>> > >>>> > >> > > > > handle the process ACTIVATED event so that the process
>> > >>>> > >> configuration
>> > >>>> > >> > is
>> > >>>> > >> > > > > added to the engine and necessary web services are
>> > created
>> > >>>> so
>> > >>>> > that
>> > >>>> > >> > when
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > load balancer send requests to any node in the
>> cluster,
>> > it
>> > >>>> is
>> > >>>> > >> ready
>> > >>>> > >> > to
>> > >>>> > >> > > > > accept those requests.
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > Regards
>> > >>>> > >> > > > > Nandika
>> > >>>> > >> > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>> > >>>> > >> sathwik.bp@gmail.com>
>> > >>>> > >> > > > > wrote:
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > > Sudharma,
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > Where are you going to configure the master-slaves,
>> is
>> > >>>> it in
>> > >>>> > the
>> > >>>> > >> > web
>> > >>>> > >> > > > > > application level or at the load balancer?
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > regards,
>> > >>>> > >> > > > > > sathwik
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma
>> subasinghe <
>> > >>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
>> > >>>> > >> > > > > > wrote:
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > > Hi Tammo,
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > Can you suggest the best method from these to
>> > >>>> implement? As
>> > >>>> > >> > first I
>> > >>>> > >> > > > > > > suggested the master-slaves scenario I think it is
>> > >>>> easy to
>> > >>>> > >> > > implement
>> > >>>> > >> > > > > than
>> > >>>> > >> > > > > > > distributed lock scenario. However if you can
>> suggest
>> > >>>> one
>> > >>>> > from
>> > >>>> > >> > > these
>> > >>>> > >> > > > > two,
>> > >>>> > >> > > > > > > then I can think about it.
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > Thank you
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
>> > >>>> sathwik.bp@gmail.com>
>> > >>>> > >> > wrote:
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > > With respect to the hotdeployment,
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > > > We can drop the deployment archive onto the
>> > >>>> deployment
>> > >>>> > >> folder.
>> > >>>> > >> > > > Since
>> > >>>> > >> > > > > > the
>> > >>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed
>> lock
>> > >>>> for
>> > >>>> > the
>> > >>>> > >> > > > > > > DeploymentUnit,
>> > >>>> > >> > > > > > > > only one of the nodes will get the lock and
>> > initiate
>> > >>>> the
>> > >>>> > >> > > > deployment.
>> > >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
>> > >>>> acquiring
>> > >>>> > the
>> > >>>> > >> > lock
>> > >>>> > >> > > > and
>> > >>>> > >> > > > > > > hence
>> > >>>> > >> > > > > > > > will silently ignore it.
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>> > >>>> > >> > > > sathwik.bp@gmail.com>
>> > >>>> > >> > > > > > > > wrote:
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > > > > Hi Tammo,
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > The distributed lock acquisition on the
>> > >>>> DeploymentUnit
>> > >>>> > >> should
>> > >>>> > >> > > be
>> > >>>> > >> > > > > > added
>> > >>>> > >> > > > > > > to
>> > >>>> > >> > > > > > > > > both DeploymentWebService and
>> DeploymentPoller.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > When a deployment operation is initiated
>> through
>> > >>>> the
>> > >>>> > >> > > > > > > > DeploymentWebService,
>> > >>>> > >> > > > > > > > > The load balancer routes it to any of the
>> > available
>> > >>>> > nodes.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
>> > >>>> acquires
>> > >>>> > the
>> > >>>> > >> > > > > Distributed
>> > >>>> > >> > > > > > > > > lock. On the remaining nodes the
>> DeploymentPoller
>> > >>>> will
>> > >>>> > >> try to
>> > >>>> > >> > > > > acquire
>> > >>>> > >> > > > > > > the
>> > >>>> > >> > > > > > > > > distributed lock and will not get it and hence
>> > will
>> > >>>> > >> silently
>> > >>>> > >> > > > ignore
>> > >>>> > >> > > > > > it.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Once the routed node completes the
>> deployment, it
>> > >>>> will
>> > >>>> > >> > release
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > > lock.
>> > >>>> > >> > > > > > > > > This way we don't have to stall the
>> > >>>> DeploymentPoller in
>> > >>>> > >> other
>> > >>>> > >> > > > > nodes.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Does it answer the concerns?
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Now, if we give the responsibility of
>> identifying
>> > >>>> the
>> > >>>> > >> master
>> > >>>> > >> > > node
>> > >>>> > >> > > > > to
>> > >>>> > >> > > > > > > the
>> > >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
>> > >>>> balancer
>> > >>>> > to
>> > >>>> > >> > > change
>> > >>>> > >> > > > > > it's
>> > >>>> > >> > > > > > > > > configuration about the master node?
>> > >>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
>> > >>>> > >> > > > > > > > > node1 -master
>> > >>>> > >> > > > > > > > > node2 - slave
>> > >>>> > >> > > > > > > > > node3 - slave
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as
>> > >>>> master
>> > >>>> > node,
>> > >>>> > >> > but
>> > >>>> > >> > > > > > > hazelcast
>> > >>>> > >> > > > > > > > > might promote Node3 as master node. They are
>> out
>> > of
>> > >>>> > sync.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Is this argument valid?
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > regards,
>> > >>>> > >> > > > > > > > > sathwik
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van
>> > Lessen <
>> > >>>> > >> > > > > > > tvanlessen@gmail.com>
>> > >>>> > >> > > > > > > > > wrote:
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > >> Hi Sudharma,
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> what do you expect from the "other nodes
>> > >>>> deployment"?
>> > >>>> > >> > > > Compilation
>> > >>>> > >> > > > > is
>> > >>>> > >> > > > > > > not
>> > >>>> > >> > > > > > > > >> needed since the CBP file is written to the
>> > >>>> (shared)
>> > >>>> > FS.
>> > >>>> > >> > > > > > Registration
>> > >>>> > >> > > > > > > is
>> > >>>> > >> > > > > > > > >> also not needed, since it is done via the
>> shared
>> > >>>> > >> database.
>> > >>>> > >> > So
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > only
>> > >>>> > >> > > > > > > > >> thing that might be needed is to tell the
>> engine
>> > >>>> that
>> > >>>> > >> there
>> > >>>> > >> > > is a
>> > >>>> > >> > > > > new
>> > >>>> > >> > > > > > > > >> deployment. I'd need to check that. If this
>> is
>> > >>>> needed,
>> > >>>> > I
>> > >>>> > >> > > revert
>> > >>>> > >> > > > my
>> > >>>> > >> > > > > > > last
>> > >>>> > >> > > > > > > > >> statement, then it is perhaps better to just
>> > send
>> > >>>> an
>> > >>>> > >> event
>> > >>>> > >> > > over
>> > >>>> > >> > > > > > > > Hazelcast
>> > >>>> > >> > > > > > > > >> to all nodes that the deployment has changed.
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> Best,
>> > >>>> > >> > > > > > > > >>   Tammo
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
>> > >>>> subasinghe <
>> > >>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
>> > >>>> > >> > > > > > > > >> > wrote:
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> > Hi Tammo,
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > The master node writes meta data. But
>> runtime
>> > >>>> > >> information
>> > >>>> > >> > > must
>> > >>>> > >> > > > > be
>> > >>>> > >> > > > > > > > >> available
>> > >>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared,
>> all
>> > >>>> nodes
>> > >>>> > will
>> > >>>> > >> > see
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > > > >> > availability of a new process. My idea is
>> for
>> > >>>> master
>> > >>>> > >> node
>> > >>>> > >> > to
>> > >>>> > >> > > > > write
>> > >>>> > >> > > > > > > the
>> > >>>> > >> > > > > > > > >> meta
>> > >>>> > >> > > > > > > > >> > data and other nodes to just read the meta
>> > data
>> > >>>> and
>> > >>>> > >> load
>> > >>>> > >> > > > > > process.So
>> > >>>> > >> > > > > > > we
>> > >>>> > >> > > > > > > > >> need
>> > >>>> > >> > > > > > > > >> > a small delay between master node
>> deployment
>> > and
>> > >>>> > other
>> > >>>> > >> > nodes
>> > >>>> > >> > > > > > > > deployment.
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > Is there anyway to set the delay between
>> > master
>> > >>>> node
>> > >>>> > >> and
>> > >>>> > >> > > > slaves
>> > >>>> > >> > > > > > > until
>> > >>>> > >> > > > > > > > >> > master node finish the deployment?
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > Thank you
>> > >>>> > >> > > > > > > > >> > Sudharma
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
>> > >>>> > >> > > > tvanlessen@gmail.com
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > > > wrote:
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > > Hi Sathwik,
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik
>> B
>> > P <
>> > >>>> > >> > > > > > > sathwik.bp@gmail.com>
>> > >>>> > >> > > > > > > > >> > wrote:
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is
>> the
>> > >>>> master
>> > >>>> > >> node
>> > >>>> > >> > in
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > > > cluster?
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > I think the easiest approach is to always
>> > >>>> elect the
>> > >>>> > >> > oldest
>> > >>>> > >> > > > > node
>> > >>>> > >> > > > > > in
>> > >>>> > >> > > > > > > > the
>> > >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast
>> > can
>> > >>>> > easily
>> > >>>> > >> > asked
>> > >>>> > >> > > > for
>> > >>>> > >> > > > > > > this
>> > >>>> > >> > > > > > > > >> > > information.
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the
>> Deployment
>> > >>>> Pollers
>> > >>>> > in
>> > >>>> > >> > the
>> > >>>> > >> > > > > slave
>> > >>>> > >> > > > > > > > nodes?
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > Absolutely.
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > Suggestion:
>> > >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
>> > >>>> Master-SLaves.
>> > >>>> > Why
>> > >>>> > >> > not
>> > >>>> > >> > > > give
>> > >>>> > >> > > > > > > every
>> > >>>> > >> > > > > > > > >> node
>> > >>>> > >> > > > > > > > >> > > in
>> > >>>> > >> > > > > > > > >> > > > the cluster the same status
>> > (Active-Active).
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > > When a new deployment is made, the load
>> > >>>> balancer
>> > >>>> > >> can
>> > >>>> > >> > > push
>> > >>>> > >> > > > it
>> > >>>> > >> > > > > > to
>> > >>>> > >> > > > > > > > any
>> > >>>> > >> > > > > > > > >> of
>> > >>>> > >> > > > > > > > >> > > the
>> > >>>> > >> > > > > > > > >> > > > available nodes. That node will
>> probably
>> > >>>> acquire
>> > >>>> > a
>> > >>>> > >> > > > > distributed
>> > >>>> > >> > > > > > > > lock
>> > >>>> > >> > > > > > > > >> on
>> > >>>> > >> > > > > > > > >> > > the
>> > >>>> > >> > > > > > > > >> > > > deployment unit and acts as master for
>> > that
>> > >>>> > >> > deployment.
>> > >>>> > >> > > > This
>> > >>>> > >> > > > > > > > ensures
>> > >>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
>> > >>>> Probably no
>> > >>>> > >> static
>> > >>>> > >> > > > > > > > >> configuration of
>> > >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor
>> in
>> > the
>> > >>>> > >> > hazelcast.
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > But this would not allow to have the
>> > >>>> hotdeployment
>> > >>>> > >> via
>> > >>>> > >> > > > > > filesystem
>> > >>>> > >> > > > > > > > >> still
>> > >>>> > >> > > > > > > > >> > > enabled, right?
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > Best,
>> > >>>> > >> > > > > > > > >> > >   Tammo
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > --
>> > >>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> --
>> > >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > >
>> > >>>> > >> > >
>> > >>>> > >> >
>> > >>>> > >>
>> > >>>> > >
>> > >>>> > >
>> > >>>> >
>> > >>>>
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Sathwik,

I will concern your approach. Thank you for your effort.

On 6 June 2015 at 18:43, Sathwik B P <sa...@gmail.com> wrote:

> I have attached to the JIRA https://issues.apache.org/jira/browse/ODE-563
>
> On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <suba.11@cse.mrt.ac.lk
> >
> wrote:
>
> > Hi Sathwik,
> >
> > I can't find the attached document. Please kind be enough to resend it.
> >
> > On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:
> >
> > > Refer this one as I have corrected the numbering of steps.
> > >
> > > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com>
> > wrote:
> > >
> > >> Hi Sudharma/Tammo,
> > >>
> > >> Kindly review attached document on my proposed approach. Let me know
> if
> > >> you have any concerns or doubts on it.
> > >>
> > >> regards,
> > >> sathwik
> > >>
> > >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com>
> > wrote:
> > >>
> > >>> find response inline
> > >>>
> > >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
> > >>> suba.11@cse.mrt.ac.lk> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> Sorry for the late reply. I was trying to achieve a proper solution.
> > >>>> Following is my approach.
> > >>>>
> > >>>> 1) I elected master node for deploying purpose. So when a master
> node
> > >>>> goes
> > >>>> down hazelcast will elect the next oldest node for master.
> > >>>>
> > >>>
> > >>>     Perfect
> > >>>
> > >>>
> > >>>> 2) ODEServer can identify whether clustering is enabled or not by
> > >>>> getting
> > >>>> the property value in ode-axis2.properties file. So I introduced new
> > >>>> property called "ode-axis2.hazelcast.clustering.enabled". If there
> is
> > no
> > >>>> clustering enabled server will work as it is. If clustering is
> > enabled,
> > >>>> cluster will be initialized.
> > >>>>
> > >>>
> > >>>     Perfect
> > >>>
> > >>>
> > >>>> 3) In manual deployment its responsibility is taken by the
> > >>>> DeploymentPoller. So I give the deployment capability to master
> node,s
> > >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to
> true.So
> > >>>> others will not be able to go into check() method in
> DeploymentPoller.
> > >>>> So
> > >>>> deployment will be done by only the master node.
> > >>>>
> > >>>
> > >>>     Perfect
> > >>>
> > >>>
> > >>>>
> > >>>> 4) In DeploymentWebService, I had to consider few cases.If the
> deploy
> > >>>> request goes to the master node, it will deploy the process through
> > web
> > >>>> service.Others pollers will not go into check() method as they are
> not
> > >>>> masters. So master can continue without any involvement of others.
> > >>>>
> > >>>>
> > >>>     Perfect
> > >>>
> > >>>
> > >>>> 5) If the deploy request goes to a slave node, it will do up to file
> > >>>> creation in the file system.Slave will be stopped at that point. As
> > only
> > >>>> master poller is checking, master can continue from created files in
> > the
> > >>>> file system.
> > >>>>
> > >>>
> > >>>     DeploymentWebService provides synchronous operations. The status
> of
> > >>> the operation should be communicated to the calling client in the
> same
> > call.
> > >>>     DeploymentPoller is a backend thread that goes over each and
> every
> > >>> directory under the Deployment directory checking for any changes to
> > >>> deploy.xml in existing processes and deploy newly added processes.
> This
> > >>> process is  sequential and time consuming. As the process directories
> > >>> grows, so does the time taken for execution of the thread.
> > >>>
> > >>>     Since the request is on a slave node and the processing is done
> on
> > >>> master node, how do you check for the completion of the
> > >>> deployment/undeployment of processes and respond back to the client
> > since
> > >>> the web service call is a synchronous operation. As DeploymentPoller
> is
> > >>> taking a lot of time in processing, your request will time out right.
> > >>>
> > >>>
> > >>>>
> > >>>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl.
> > Each
> > >>>> _deploymentUnits stores only what its server has deployed. So think,
> > >>>> that a
> > >>>> master node goes down another master node appears.But its
> > >>>> __deploymentUnits
> > >>>> does not have dus which has deployed by the earlier master node.
> Hence
> > >>>> it
> > >>>> will not be able retire earlier version of the process which is
> > >>>> deployed by
> > >>>> previous master. So there will two process which are in "ACTIVE"
> state
> > >>>
> > >>>
> > >>>> 7) To avoid this, I add the ODEServer as an Observer to check when a
> > new
> > >>>> master is electing, then load all the deployment units out of the
> > >>>> store. So
> > >>>> new master node can have all the dus and can retire appropriate
> > version.
> > >>>> Usually loadAll() is called at the server start-up time. But there
> is
> > no
> > >>>> other way to solve this. I tried to use Hazelcast IMap to store all
> > dus
> > >>>> among all nodes. But it wasn't success as du is not serializable
> > >>>> object.
> > >>>>
> > >>>
> > >>>> 8) I figured out that we do not need send cluster message to others
> as
> > >>>> all
> > >>>> the dus' data are persisted to the shared DB. So each node can take
> > the
> > >>>> du
> > >>>> and retrieve necessary data using already implemented methods in
> > Process
> > >>>> Store.
> > >>>>
> > >>>> 9) But there is an another problem.The axis2 service corresponding
> to
> > a
> > >>>> deployed process does not appear on all nodes of the cluster. That
> is
> > >>>> because each server add du which is deployed by it to the process
> > >>>> store.That is why I had use loadAll() when masters are changing. How
> > to
> > >>>> solve this?
> > >>>>
> > >>>
> > >>>     I do appriciate your efforts in understanding the implmentation
> and
> > >>> changes that need to be done. You are bang on it.
> > >>>     fireEvent(..) is the method that triggers process activation and
> > >>> necessary service creation.
> > >>>
> > >>>
> > >>>     But with this given apporach from steps 6 to 8, ODE cannot have
> > >>> atleast 2 Active servers for Load balancing. You are concentrating on
> > only
> > >>> one active node that will do deployments and cater to process
> > invocations.
> > >>>     We should also think about scaling ODE to multiple servers to
> > handle
> > >>> load.
> > >>>
> > >>>     What do you think.
> > >>>
> > >>>
> > >>>> Thank you,
> > >>>> Sudharma
> > >>>>
> > >>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
> > >>>>
> > >>>> > Sudharma,
> > >>>> >
> > >>>> > Any updates?
> > >>>> >
> > >>>> > regards,
> > >>>> > sathwik
> > >>>> >
> > >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <
> sathwik.bp@gmail.com>
> > >>>> wrote:
> > >>>> >
> > >>>> > > Sudharma,
> > >>>> > >
> > >>>> > > Can you elaborate on your option 1).
> > >>>> > >
> > >>>> > > Response to your option 2).
> > >>>> > >
> > >>>> > >     Process Store is the component that handles process
> metadata,
> > >>>> > > compilation and deployment in ODE. Integration layers in ODE
> > >>>> (Axis2, JBI)
> > >>>> > > use the process store.
> > >>>> > >     Future implementations of IL for ODE will also use the
> process
> > >>>> store.
> > >>>> > > We should not be thinking of moving the process store
> > functionality
> > >>>> to
> > >>>> > the
> > >>>> > > integration layers.
> > >>>> > >
> > >>>> > >
> > >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> > >>>> > > suba.11@cse.mrt.ac.lk> wrote:
> > >>>> > >
> > >>>> > >> Hi,
> > >>>> > >>
> > >>>> > >> I understood the problem within dynamic master/slave
> > >>>> configuration. In
> > >>>> > my
> > >>>> > >> approach, when a deployment request is routed to a slave node
> > >>>> there will
> > >>>> > >> not be a deployment. I suggest two options to avoid it.
> > >>>> > >> 1) Have static master/slave configuration only for deploy
> process
> > >>>> > >>
> > >>>> > > 2) Modify the deployment web service to complie and verify the
> > >>>> process
> > >>>> > and
> > >>>> > >> then copy it to the deploy folder irrespective of whether its a
> > >>>> master
> > >>>> > or
> > >>>> > >> slave, then deployment poller should take care of the
> deployment
> > >>>> > >>
> > >>>> > >>
> > >>>> > >
> > >>>> > >>
> > >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com>
> > wrote:
> > >>>> > >>
> > >>>> > >> > Sudharma,
> > >>>> > >> >
> > >>>> > >> > We definitely need a master/slave in the hazelcast cluster.
> > This
> > >>>> is
> > >>>> > >> > probably needed for the job migration in the Scheduler to
> > >>>> migrate the
> > >>>> > >> jobs
> > >>>> > >> > associated with a down node. Let hold on this topic for
> future
> > >>>> > >> discussion.
> > >>>> > >> >
> > >>>> > >> > Going by the explanation where the master/slave nodes have
> > >>>> certain
> > >>>> > >> > predefined tasks to perform is perfectly fine.
> > >>>> > >> >
> > >>>> > >> > I have this scenario,
> > >>>> > >> >
> > >>>> > >> > I am using HAProxy as my load balancer and configured 3 nodes
> > in
> > >>>> the
> > >>>> > >> > cluster.
> > >>>> > >> >
> > >>>> > >> > Node1 - Active
> > >>>> > >> > Node2 - Active
> > >>>> > >> > Node3 - Backup
> > >>>> > >> >
> > >>>> > >> > Load balancing algorithm: RoundRobin
> > >>>> > >> >
> > >>>> > >> > A Backup node (Node3) is one which the load balancer will not
> > >>>> route
> > >>>> > >> > requests to, until one of the Active node i.e either Node1 or
> > >>>> Node2
> > >>>> > has
> > >>>> > >> > gone down.
> > >>>> > >> >
> > >>>> > >> > All these 3 nodes are also part of the hazelcast cluster as
> > well.
> > >>>> > >> >
> > >>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
> > >>>> leader/master
> > >>>> > >> and
> > >>>> > >> > Node2,Node3 as slaves.
> > >>>> > >> >
> > >>>> > >> > I initiate the deploy operation on the DeploymentWebService
> > >>>> which the
> > >>>> > >> load
> > >>>> > >> > balancer routes it to one of the Active nodes in the cluster,
> > >>>> lets say
> > >>>> > >> it's
> > >>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
> > >>>> cluster,
> > >>>> > >> > deployment is a success.
> > >>>> > >> >
> > >>>> > >> > I initiate another deploy operation on the
> DeploymentWebService
> > >>>> which
> > >>>> > >> the
> > >>>> > >> > load balancer routes it to the next active node which is
> Node2.
> > >>>> Since
> > >>>> > >> Node2
> > >>>> > >> > is a slave in the Hazelcast cluster, What happens to the
> > >>>> deployment?
> > >>>> > >> >
> > >>>> > >> > regards,
> > >>>> > >> > sathwik
> > >>>> > >> >
> > >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> > >>>> > >> > suba.11@cse.mrt.ac.lk
> > >>>> > >> > > wrote:
> > >>>> > >> >
> > >>>> > >> > > Hi,
> > >>>> > >> > >
> > >>>> > >> > > I will explain my approach as much as possible. The oldest
> > >>>> node in
> > >>>> > the
> > >>>> > >> > > hazelcast cluster is elected as the master node. In the
> > >>>> failure of
> > >>>> > the
> > >>>> > >> > > master node, next oldest node will be elected as the master
> > >>>> node.
> > >>>> > This
> > >>>> > >> > > master-slave configuration is just for deployment. When the
> > >>>> > hazelcast
> > >>>> > >> > > cluster elected the master node, that node becomes a master
> > >>>> node for
> > >>>> > >> > > deploying process. So it will do the deploying artifacts.
> If
> > >>>> you
> > >>>> > want
> > >>>> > >> to
> > >>>> > >> > > get the idea of electing master node please refer the code
> > >>>> which I
> > >>>> > >> have
> > >>>> > >> > > located in the github. (
> > >>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
> > >>>> > >> > >
> > >>>> > >> > > I identified separated actions which should be followed by
> > the
> > >>>> > master
> > >>>> > >> and
> > >>>> > >> > > salve nodes.
> > >>>> > >> > > Actions which are followed by master node only
> > >>>> > >> > > 1) create deployment unit
> > >>>> > >> > > 2) set the version nu to deployment unit
> > >>>> > >> > > 3) compile deployment unit
> > >>>> > >> > > 4) scan deployment unit
> > >>>> > >> > > 5) retire previous versions
> > >>>> > >> > > Master node and slave nodes should create _processes which
> > >>>> stores
> > >>>> > >> > > ProcessConfImpl
> > >>>> > >> > > Only master node will write the version nu to database,
> > create
> > >>>> > >> .deployed
> > >>>> > >> > > file
> > >>>> > >> > >
> > >>>> > >> > > So there are some actions which should be followed only by
> > >>>> master
> > >>>> > node
> > >>>> > >> > > while other actions should be followed by all the nodes.The
> > >>>> idea of
> > >>>> > >> > having
> > >>>> > >> > > a master node is deploying artifacts and avoid others from
> > >>>> writing
> > >>>> > the
> > >>>> > >> > > version nu to database.
> > >>>> > >> > > Whether a node is active or passive, all nodes should do
> the
> > >>>> > >> > > deployment.Master
> > >>>> > >> > > and slaves will follow necessary actions as in above.
> > >>>> > >> > >
> > >>>> > >> > >
> > >>>> > >> > >
> > >>>> > >> > >
> > >>>> > >> > >
> > >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <sathwik.bp@gmail.com
> >
> > >>>> wrote:
> > >>>> > >> > >
> > >>>> > >> > > > Nandika,
> > >>>> > >> > > >
> > >>>> > >> > > > I very well understand what you have put across, but it's
> > >>>> > secondary
> > >>>> > >> to
> > >>>> > >> > me
> > >>>> > >> > > > now.
> > >>>> > >> > > >
> > >>>> > >> > > > Sudharma,
> > >>>> > >> > > > My primary concern is to understand at a high level the
> > >>>> deployment
> > >>>> > >> > > > architecture and how would master-slave configuration fit
> > >>>> in. Are
> > >>>> > >> there
> > >>>> > >> > > any
> > >>>> > >> > > > restrictions imposed by the in-progress design?
> > >>>> > >> > > >
> > >>>> > >> > > > Firstly, how would ODE process deployment work under
> these
> > >>>> cluster
> > >>>> > >> > > > configurations?
> > >>>> > >> > > >
> > >>>> > >> > > > Sample Cluster configurations: A load balancer is
> > >>>> frontending the
> > >>>> > >> > > servers.
> > >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> > >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> > >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
> > >>>> Active or
> > >>>> > >> > > Passive.
> > >>>> > >> > > >
> > >>>> > >> > > > regards,
> > >>>> > >> > > > sathwik
> > >>>> > >> > > >
> > >>>> > >> > > >
> > >>>> > >> > > >
> > >>>> > >> > > >
> > >>>> > >> > > >
> > >>>> > >> > > >
> > >>>> > >> > > >
> > >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> > >>>> > >> > jayawark@gmail.com
> > >>>> > >> > > >
> > >>>> > >> > > > wrote:
> > >>>> > >> > > >
> > >>>> > >> > > > > Hi Sathwik,
> > >>>> > >> > > > >
> > >>>> > >> > > > > According to my understanding, in the clustering
> > scenario,
> > >>>> the
> > >>>> > >> master
> > >>>> > >> > > > node
> > >>>> > >> > > > > should perform all the deployment actions and the slave
> > >>>> nodes
> > >>>> > also
> > >>>> > >> > need
> > >>>> > >> > > > to
> > >>>> > >> > > > > perform some deployment actions. For example, the slave
> > >>>> nodes
> > >>>> > also
> > >>>> > >> > > should
> > >>>> > >> > > > > handle the process ACTIVATED event so that the process
> > >>>> > >> configuration
> > >>>> > >> > is
> > >>>> > >> > > > > added to the engine and necessary web services are
> > created
> > >>>> so
> > >>>> > that
> > >>>> > >> > when
> > >>>> > >> > > > the
> > >>>> > >> > > > > load balancer send requests to any node in the cluster,
> > it
> > >>>> is
> > >>>> > >> ready
> > >>>> > >> > to
> > >>>> > >> > > > > accept those requests.
> > >>>> > >> > > > >
> > >>>> > >> > > > > Regards
> > >>>> > >> > > > > Nandika
> > >>>> > >> > > > >
> > >>>> > >> > > > >
> > >>>> > >> > > > >
> > >>>> > >> > > > >
> > >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> > >>>> > >> sathwik.bp@gmail.com>
> > >>>> > >> > > > > wrote:
> > >>>> > >> > > > >
> > >>>> > >> > > > > > Sudharma,
> > >>>> > >> > > > > >
> > >>>> > >> > > > > > Where are you going to configure the master-slaves,
> is
> > >>>> it in
> > >>>> > the
> > >>>> > >> > web
> > >>>> > >> > > > > > application level or at the load balancer?
> > >>>> > >> > > > > >
> > >>>> > >> > > > > > regards,
> > >>>> > >> > > > > > sathwik
> > >>>> > >> > > > > >
> > >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe
> <
> > >>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
> > >>>> > >> > > > > > wrote:
> > >>>> > >> > > > > >
> > >>>> > >> > > > > > > Hi Tammo,
> > >>>> > >> > > > > > >
> > >>>> > >> > > > > > > Can you suggest the best method from these to
> > >>>> implement? As
> > >>>> > >> > first I
> > >>>> > >> > > > > > > suggested the master-slaves scenario I think it is
> > >>>> easy to
> > >>>> > >> > > implement
> > >>>> > >> > > > > than
> > >>>> > >> > > > > > > distributed lock scenario. However if you can
> suggest
> > >>>> one
> > >>>> > from
> > >>>> > >> > > these
> > >>>> > >> > > > > two,
> > >>>> > >> > > > > > > then I can think about it.
> > >>>> > >> > > > > > >
> > >>>> > >> > > > > > > Thank you
> > >>>> > >> > > > > > >
> > >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
> > >>>> sathwik.bp@gmail.com>
> > >>>> > >> > wrote:
> > >>>> > >> > > > > > >
> > >>>> > >> > > > > > > > With respect to the hotdeployment,
> > >>>> > >> > > > > > > >
> > >>>> > >> > > > > > > > We can drop the deployment archive onto the
> > >>>> deployment
> > >>>> > >> folder.
> > >>>> > >> > > > Since
> > >>>> > >> > > > > > the
> > >>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed
> lock
> > >>>> for
> > >>>> > the
> > >>>> > >> > > > > > > DeploymentUnit,
> > >>>> > >> > > > > > > > only one of the nodes will get the lock and
> > initiate
> > >>>> the
> > >>>> > >> > > > deployment.
> > >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
> > >>>> acquiring
> > >>>> > the
> > >>>> > >> > lock
> > >>>> > >> > > > and
> > >>>> > >> > > > > > > hence
> > >>>> > >> > > > > > > > will silently ignore it.
> > >>>> > >> > > > > > > >
> > >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> > >>>> > >> > > > sathwik.bp@gmail.com>
> > >>>> > >> > > > > > > > wrote:
> > >>>> > >> > > > > > > >
> > >>>> > >> > > > > > > > > Hi Tammo,
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > The distributed lock acquisition on the
> > >>>> DeploymentUnit
> > >>>> > >> should
> > >>>> > >> > > be
> > >>>> > >> > > > > > added
> > >>>> > >> > > > > > > to
> > >>>> > >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > When a deployment operation is initiated
> through
> > >>>> the
> > >>>> > >> > > > > > > > DeploymentWebService,
> > >>>> > >> > > > > > > > > The load balancer routes it to any of the
> > available
> > >>>> > nodes.
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
> > >>>> acquires
> > >>>> > the
> > >>>> > >> > > > > Distributed
> > >>>> > >> > > > > > > > > lock. On the remaining nodes the
> DeploymentPoller
> > >>>> will
> > >>>> > >> try to
> > >>>> > >> > > > > acquire
> > >>>> > >> > > > > > > the
> > >>>> > >> > > > > > > > > distributed lock and will not get it and hence
> > will
> > >>>> > >> silently
> > >>>> > >> > > > ignore
> > >>>> > >> > > > > > it.
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > Once the routed node completes the deployment,
> it
> > >>>> will
> > >>>> > >> > release
> > >>>> > >> > > > the
> > >>>> > >> > > > > > > lock.
> > >>>> > >> > > > > > > > > This way we don't have to stall the
> > >>>> DeploymentPoller in
> > >>>> > >> other
> > >>>> > >> > > > > nodes.
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > Does it answer the concerns?
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > Now, if we give the responsibility of
> identifying
> > >>>> the
> > >>>> > >> master
> > >>>> > >> > > node
> > >>>> > >> > > > > to
> > >>>> > >> > > > > > > the
> > >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
> > >>>> balancer
> > >>>> > to
> > >>>> > >> > > change
> > >>>> > >> > > > > > it's
> > >>>> > >> > > > > > > > > configuration about the master node?
> > >>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
> > >>>> > >> > > > > > > > > node1 -master
> > >>>> > >> > > > > > > > > node2 - slave
> > >>>> > >> > > > > > > > > node3 - slave
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as
> > >>>> master
> > >>>> > node,
> > >>>> > >> > but
> > >>>> > >> > > > > > > hazelcast
> > >>>> > >> > > > > > > > > might promote Node3 as master node. They are
> out
> > of
> > >>>> > sync.
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > Is this argument valid?
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > regards,
> > >>>> > >> > > > > > > > > sathwik
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van
> > Lessen <
> > >>>> > >> > > > > > > tvanlessen@gmail.com>
> > >>>> > >> > > > > > > > > wrote:
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > >> Hi Sudharma,
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >> what do you expect from the "other nodes
> > >>>> deployment"?
> > >>>> > >> > > > Compilation
> > >>>> > >> > > > > is
> > >>>> > >> > > > > > > not
> > >>>> > >> > > > > > > > >> needed since the CBP file is written to the
> > >>>> (shared)
> > >>>> > FS.
> > >>>> > >> > > > > > Registration
> > >>>> > >> > > > > > > is
> > >>>> > >> > > > > > > > >> also not needed, since it is done via the
> shared
> > >>>> > >> database.
> > >>>> > >> > So
> > >>>> > >> > > > the
> > >>>> > >> > > > > > only
> > >>>> > >> > > > > > > > >> thing that might be needed is to tell the
> engine
> > >>>> that
> > >>>> > >> there
> > >>>> > >> > > is a
> > >>>> > >> > > > > new
> > >>>> > >> > > > > > > > >> deployment. I'd need to check that. If this is
> > >>>> needed,
> > >>>> > I
> > >>>> > >> > > revert
> > >>>> > >> > > > my
> > >>>> > >> > > > > > > last
> > >>>> > >> > > > > > > > >> statement, then it is perhaps better to just
> > send
> > >>>> an
> > >>>> > >> event
> > >>>> > >> > > over
> > >>>> > >> > > > > > > > Hazelcast
> > >>>> > >> > > > > > > > >> to all nodes that the deployment has changed.
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >> Best,
> > >>>> > >> > > > > > > > >>   Tammo
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
> > >>>> subasinghe <
> > >>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
> > >>>> > >> > > > > > > > >> > wrote:
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >> > Hi Tammo,
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >> > The master node writes meta data. But
> runtime
> > >>>> > >> information
> > >>>> > >> > > must
> > >>>> > >> > > > > be
> > >>>> > >> > > > > > > > >> available
> > >>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared, all
> > >>>> nodes
> > >>>> > will
> > >>>> > >> > see
> > >>>> > >> > > > the
> > >>>> > >> > > > > > > > >> > availability of a new process. My idea is
> for
> > >>>> master
> > >>>> > >> node
> > >>>> > >> > to
> > >>>> > >> > > > > write
> > >>>> > >> > > > > > > the
> > >>>> > >> > > > > > > > >> meta
> > >>>> > >> > > > > > > > >> > data and other nodes to just read the meta
> > data
> > >>>> and
> > >>>> > >> load
> > >>>> > >> > > > > > process.So
> > >>>> > >> > > > > > > we
> > >>>> > >> > > > > > > > >> need
> > >>>> > >> > > > > > > > >> > a small delay between master node deployment
> > and
> > >>>> > other
> > >>>> > >> > nodes
> > >>>> > >> > > > > > > > deployment.
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >> > Is there anyway to set the delay between
> > master
> > >>>> node
> > >>>> > >> and
> > >>>> > >> > > > slaves
> > >>>> > >> > > > > > > until
> > >>>> > >> > > > > > > > >> > master node finish the deployment?
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >> > Thank you
> > >>>> > >> > > > > > > > >> > Sudharma
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> > >>>> > >> > > > tvanlessen@gmail.com
> > >>>> > >> > > > > >
> > >>>> > >> > > > > > > > wrote:
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >> > > Hi Sathwik,
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B
> > P <
> > >>>> > >> > > > > > > sathwik.bp@gmail.com>
> > >>>> > >> > > > > > > > >> > wrote:
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
> > >>>> > >> > > > > > > > >> > > >
> > >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is the
> > >>>> master
> > >>>> > >> node
> > >>>> > >> > in
> > >>>> > >> > > > the
> > >>>> > >> > > > > > > > cluster?
> > >>>> > >> > > > > > > > >> > > >
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > I think the easiest approach is to always
> > >>>> elect the
> > >>>> > >> > oldest
> > >>>> > >> > > > > node
> > >>>> > >> > > > > > in
> > >>>> > >> > > > > > > > the
> > >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast
> > can
> > >>>> > easily
> > >>>> > >> > asked
> > >>>> > >> > > > for
> > >>>> > >> > > > > > > this
> > >>>> > >> > > > > > > > >> > > information.
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment
> > >>>> Pollers
> > >>>> > in
> > >>>> > >> > the
> > >>>> > >> > > > > slave
> > >>>> > >> > > > > > > > nodes?
> > >>>> > >> > > > > > > > >> > > >
> > >>>> > >> > > > > > > > >> > > >
> > >>>> > >> > > > > > > > >> > > Absolutely.
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > Suggestion:
> > >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
> > >>>> Master-SLaves.
> > >>>> > Why
> > >>>> > >> > not
> > >>>> > >> > > > give
> > >>>> > >> > > > > > > every
> > >>>> > >> > > > > > > > >> node
> > >>>> > >> > > > > > > > >> > > in
> > >>>> > >> > > > > > > > >> > > > the cluster the same status
> > (Active-Active).
> > >>>> > >> > > > > > > > >> > > >
> > >>>> > >> > > > > > > > >> > > > When a new deployment is made, the load
> > >>>> balancer
> > >>>> > >> can
> > >>>> > >> > > push
> > >>>> > >> > > > it
> > >>>> > >> > > > > > to
> > >>>> > >> > > > > > > > any
> > >>>> > >> > > > > > > > >> of
> > >>>> > >> > > > > > > > >> > > the
> > >>>> > >> > > > > > > > >> > > > available nodes. That node will probably
> > >>>> acquire
> > >>>> > a
> > >>>> > >> > > > > distributed
> > >>>> > >> > > > > > > > lock
> > >>>> > >> > > > > > > > >> on
> > >>>> > >> > > > > > > > >> > > the
> > >>>> > >> > > > > > > > >> > > > deployment unit and acts as master for
> > that
> > >>>> > >> > deployment.
> > >>>> > >> > > > This
> > >>>> > >> > > > > > > > ensures
> > >>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
> > >>>> Probably no
> > >>>> > >> static
> > >>>> > >> > > > > > > > >> configuration of
> > >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor in
> > the
> > >>>> > >> > hazelcast.
> > >>>> > >> > > > > > > > >> > > >
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > But this would not allow to have the
> > >>>> hotdeployment
> > >>>> > >> via
> > >>>> > >> > > > > > filesystem
> > >>>> > >> > > > > > > > >> still
> > >>>> > >> > > > > > > > >> > > enabled, right?
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > Best,
> > >>>> > >> > > > > > > > >> > >   Tammo
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> > > --
> > >>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> > >>>> > >> > > > > > > > >> > >
> > >>>> > >> > > > > > > > >> >
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >> --
> > >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> > >>>> > >> > > > > > > > >>
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > > >
> > >>>> > >> > > > > > > >
> > >>>> > >> > > > > > >
> > >>>> > >> > > > > >
> > >>>> > >> > > > >
> > >>>> > >> > > >
> > >>>> > >> > >
> > >>>> > >> >
> > >>>> > >>
> > >>>> > >
> > >>>> > >
> > >>>> >
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

I have attached to the JIRA https://issues.apache.org/jira/browse/ODE-563

On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <su...@cse.mrt.ac.lk>
wrote:

> Hi Sathwik,
>
> I can't find the attached document. Please kind be enough to resend it.
>
> On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:
>
> > Refer this one as I have corrected the numbering of steps.
> >
> > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com>
> wrote:
> >
> >> Hi Sudharma/Tammo,
> >>
> >> Kindly review attached document on my proposed approach. Let me know if
> >> you have any concerns or doubts on it.
> >>
> >> regards,
> >> sathwik
> >>
> >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com>
> wrote:
> >>
> >>> find response inline
> >>>
> >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
> >>> suba.11@cse.mrt.ac.lk> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Sorry for the late reply. I was trying to achieve a proper solution.
> >>>> Following is my approach.
> >>>>
> >>>> 1) I elected master node for deploying purpose. So when a master node
> >>>> goes
> >>>> down hazelcast will elect the next oldest node for master.
> >>>>
> >>>
> >>>     Perfect
> >>>
> >>>
> >>>> 2) ODEServer can identify whether clustering is enabled or not by
> >>>> getting
> >>>> the property value in ode-axis2.properties file. So I introduced new
> >>>> property called "ode-axis2.hazelcast.clustering.enabled". If there is
> no
> >>>> clustering enabled server will work as it is. If clustering is
> enabled,
> >>>> cluster will be initialized.
> >>>>
> >>>
> >>>     Perfect
> >>>
> >>>
> >>>> 3) In manual deployment its responsibility is taken by the
> >>>> DeploymentPoller. So I give the deployment capability to master node,s
> >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to true.So
> >>>> others will not be able to go into check() method in DeploymentPoller.
> >>>> So
> >>>> deployment will be done by only the master node.
> >>>>
> >>>
> >>>     Perfect
> >>>
> >>>
> >>>>
> >>>> 4) In DeploymentWebService, I had to consider few cases.If the deploy
> >>>> request goes to the master node, it will deploy the process through
> web
> >>>> service.Others pollers will not go into check() method as they are not
> >>>> masters. So master can continue without any involvement of others.
> >>>>
> >>>>
> >>>     Perfect
> >>>
> >>>
> >>>> 5) If the deploy request goes to a slave node, it will do up to file
> >>>> creation in the file system.Slave will be stopped at that point. As
> only
> >>>> master poller is checking, master can continue from created files in
> the
> >>>> file system.
> >>>>
> >>>
> >>>     DeploymentWebService provides synchronous operations. The status of
> >>> the operation should be communicated to the calling client in the same
> call.
> >>>     DeploymentPoller is a backend thread that goes over each and every
> >>> directory under the Deployment directory checking for any changes to
> >>> deploy.xml in existing processes and deploy newly added processes. This
> >>> process is  sequential and time consuming. As the process directories
> >>> grows, so does the time taken for execution of the thread.
> >>>
> >>>     Since the request is on a slave node and the processing is done on
> >>> master node, how do you check for the completion of the
> >>> deployment/undeployment of processes and respond back to the client
> since
> >>> the web service call is a synchronous operation. As DeploymentPoller is
> >>> taking a lot of time in processing, your request will time out right.
> >>>
> >>>
> >>>>
> >>>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl.
> Each
> >>>> _deploymentUnits stores only what its server has deployed. So think,
> >>>> that a
> >>>> master node goes down another master node appears.But its
> >>>> __deploymentUnits
> >>>> does not have dus which has deployed by the earlier master node. Hence
> >>>> it
> >>>> will not be able retire earlier version of the process which is
> >>>> deployed by
> >>>> previous master. So there will two process which are in "ACTIVE" state
> >>>
> >>>
> >>>> 7) To avoid this, I add the ODEServer as an Observer to check when a
> new
> >>>> master is electing, then load all the deployment units out of the
> >>>> store. So
> >>>> new master node can have all the dus and can retire appropriate
> version.
> >>>> Usually loadAll() is called at the server start-up time. But there is
> no
> >>>> other way to solve this. I tried to use Hazelcast IMap to store all
> dus
> >>>> among all nodes. But it wasn't success as du is not serializable
> >>>> object.
> >>>>
> >>>
> >>>> 8) I figured out that we do not need send cluster message to others as
> >>>> all
> >>>> the dus' data are persisted to the shared DB. So each node can take
> the
> >>>> du
> >>>> and retrieve necessary data using already implemented methods in
> Process
> >>>> Store.
> >>>>
> >>>> 9) But there is an another problem.The axis2 service corresponding to
> a
> >>>> deployed process does not appear on all nodes of the cluster. That is
> >>>> because each server add du which is deployed by it to the process
> >>>> store.That is why I had use loadAll() when masters are changing. How
> to
> >>>> solve this?
> >>>>
> >>>
> >>>     I do appriciate your efforts in understanding the implmentation and
> >>> changes that need to be done. You are bang on it.
> >>>     fireEvent(..) is the method that triggers process activation and
> >>> necessary service creation.
> >>>
> >>>
> >>>     But with this given apporach from steps 6 to 8, ODE cannot have
> >>> atleast 2 Active servers for Load balancing. You are concentrating on
> only
> >>> one active node that will do deployments and cater to process
> invocations.
> >>>     We should also think about scaling ODE to multiple servers to
> handle
> >>> load.
> >>>
> >>>     What do you think.
> >>>
> >>>
> >>>> Thank you,
> >>>> Sudharma
> >>>>
> >>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
> >>>>
> >>>> > Sudharma,
> >>>> >
> >>>> > Any updates?
> >>>> >
> >>>> > regards,
> >>>> > sathwik
> >>>> >
> >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com>
> >>>> wrote:
> >>>> >
> >>>> > > Sudharma,
> >>>> > >
> >>>> > > Can you elaborate on your option 1).
> >>>> > >
> >>>> > > Response to your option 2).
> >>>> > >
> >>>> > >     Process Store is the component that handles process metadata,
> >>>> > > compilation and deployment in ODE. Integration layers in ODE
> >>>> (Axis2, JBI)
> >>>> > > use the process store.
> >>>> > >     Future implementations of IL for ODE will also use the process
> >>>> store.
> >>>> > > We should not be thinking of moving the process store
> functionality
> >>>> to
> >>>> > the
> >>>> > > integration layers.
> >>>> > >
> >>>> > >
> >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> >>>> > > suba.11@cse.mrt.ac.lk> wrote:
> >>>> > >
> >>>> > >> Hi,
> >>>> > >>
> >>>> > >> I understood the problem within dynamic master/slave
> >>>> configuration. In
> >>>> > my
> >>>> > >> approach, when a deployment request is routed to a slave node
> >>>> there will
> >>>> > >> not be a deployment. I suggest two options to avoid it.
> >>>> > >> 1) Have static master/slave configuration only for deploy process
> >>>> > >>
> >>>> > > 2) Modify the deployment web service to complie and verify the
> >>>> process
> >>>> > and
> >>>> > >> then copy it to the deploy folder irrespective of whether its a
> >>>> master
> >>>> > or
> >>>> > >> slave, then deployment poller should take care of the deployment
> >>>> > >>
> >>>> > >>
> >>>> > >
> >>>> > >>
> >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com>
> wrote:
> >>>> > >>
> >>>> > >> > Sudharma,
> >>>> > >> >
> >>>> > >> > We definitely need a master/slave in the hazelcast cluster.
> This
> >>>> is
> >>>> > >> > probably needed for the job migration in the Scheduler to
> >>>> migrate the
> >>>> > >> jobs
> >>>> > >> > associated with a down node. Let hold on this topic for future
> >>>> > >> discussion.
> >>>> > >> >
> >>>> > >> > Going by the explanation where the master/slave nodes have
> >>>> certain
> >>>> > >> > predefined tasks to perform is perfectly fine.
> >>>> > >> >
> >>>> > >> > I have this scenario,
> >>>> > >> >
> >>>> > >> > I am using HAProxy as my load balancer and configured 3 nodes
> in
> >>>> the
> >>>> > >> > cluster.
> >>>> > >> >
> >>>> > >> > Node1 - Active
> >>>> > >> > Node2 - Active
> >>>> > >> > Node3 - Backup
> >>>> > >> >
> >>>> > >> > Load balancing algorithm: RoundRobin
> >>>> > >> >
> >>>> > >> > A Backup node (Node3) is one which the load balancer will not
> >>>> route
> >>>> > >> > requests to, until one of the Active node i.e either Node1 or
> >>>> Node2
> >>>> > has
> >>>> > >> > gone down.
> >>>> > >> >
> >>>> > >> > All these 3 nodes are also part of the hazelcast cluster as
> well.
> >>>> > >> >
> >>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
> >>>> leader/master
> >>>> > >> and
> >>>> > >> > Node2,Node3 as slaves.
> >>>> > >> >
> >>>> > >> > I initiate the deploy operation on the DeploymentWebService
> >>>> which the
> >>>> > >> load
> >>>> > >> > balancer routes it to one of the Active nodes in the cluster,
> >>>> lets say
> >>>> > >> it's
> >>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
> >>>> cluster,
> >>>> > >> > deployment is a success.
> >>>> > >> >
> >>>> > >> > I initiate another deploy operation on the DeploymentWebService
> >>>> which
> >>>> > >> the
> >>>> > >> > load balancer routes it to the next active node which is Node2.
> >>>> Since
> >>>> > >> Node2
> >>>> > >> > is a slave in the Hazelcast cluster, What happens to the
> >>>> deployment?
> >>>> > >> >
> >>>> > >> > regards,
> >>>> > >> > sathwik
> >>>> > >> >
> >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> >>>> > >> > suba.11@cse.mrt.ac.lk
> >>>> > >> > > wrote:
> >>>> > >> >
> >>>> > >> > > Hi,
> >>>> > >> > >
> >>>> > >> > > I will explain my approach as much as possible. The oldest
> >>>> node in
> >>>> > the
> >>>> > >> > > hazelcast cluster is elected as the master node. In the
> >>>> failure of
> >>>> > the
> >>>> > >> > > master node, next oldest node will be elected as the master
> >>>> node.
> >>>> > This
> >>>> > >> > > master-slave configuration is just for deployment. When the
> >>>> > hazelcast
> >>>> > >> > > cluster elected the master node, that node becomes a master
> >>>> node for
> >>>> > >> > > deploying process. So it will do the deploying artifacts. If
> >>>> you
> >>>> > want
> >>>> > >> to
> >>>> > >> > > get the idea of electing master node please refer the code
> >>>> which I
> >>>> > >> have
> >>>> > >> > > located in the github. (
> >>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
> >>>> > >> > >
> >>>> > >> > > I identified separated actions which should be followed by
> the
> >>>> > master
> >>>> > >> and
> >>>> > >> > > salve nodes.
> >>>> > >> > > Actions which are followed by master node only
> >>>> > >> > > 1) create deployment unit
> >>>> > >> > > 2) set the version nu to deployment unit
> >>>> > >> > > 3) compile deployment unit
> >>>> > >> > > 4) scan deployment unit
> >>>> > >> > > 5) retire previous versions
> >>>> > >> > > Master node and slave nodes should create _processes which
> >>>> stores
> >>>> > >> > > ProcessConfImpl
> >>>> > >> > > Only master node will write the version nu to database,
> create
> >>>> > >> .deployed
> >>>> > >> > > file
> >>>> > >> > >
> >>>> > >> > > So there are some actions which should be followed only by
> >>>> master
> >>>> > node
> >>>> > >> > > while other actions should be followed by all the nodes.The
> >>>> idea of
> >>>> > >> > having
> >>>> > >> > > a master node is deploying artifacts and avoid others from
> >>>> writing
> >>>> > the
> >>>> > >> > > version nu to database.
> >>>> > >> > > Whether a node is active or passive, all nodes should do the
> >>>> > >> > > deployment.Master
> >>>> > >> > > and slaves will follow necessary actions as in above.
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com>
> >>>> wrote:
> >>>> > >> > >
> >>>> > >> > > > Nandika,
> >>>> > >> > > >
> >>>> > >> > > > I very well understand what you have put across, but it's
> >>>> > secondary
> >>>> > >> to
> >>>> > >> > me
> >>>> > >> > > > now.
> >>>> > >> > > >
> >>>> > >> > > > Sudharma,
> >>>> > >> > > > My primary concern is to understand at a high level the
> >>>> deployment
> >>>> > >> > > > architecture and how would master-slave configuration fit
> >>>> in. Are
> >>>> > >> there
> >>>> > >> > > any
> >>>> > >> > > > restrictions imposed by the in-progress design?
> >>>> > >> > > >
> >>>> > >> > > > Firstly, how would ODE process deployment work under these
> >>>> cluster
> >>>> > >> > > > configurations?
> >>>> > >> > > >
> >>>> > >> > > > Sample Cluster configurations: A load balancer is
> >>>> frontending the
> >>>> > >> > > servers.
> >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
> >>>> Active or
> >>>> > >> > > Passive.
> >>>> > >> > > >
> >>>> > >> > > > regards,
> >>>> > >> > > > sathwik
> >>>> > >> > > >
> >>>> > >> > > >
> >>>> > >> > > >
> >>>> > >> > > >
> >>>> > >> > > >
> >>>> > >> > > >
> >>>> > >> > > >
> >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> >>>> > >> > jayawark@gmail.com
> >>>> > >> > > >
> >>>> > >> > > > wrote:
> >>>> > >> > > >
> >>>> > >> > > > > Hi Sathwik,
> >>>> > >> > > > >
> >>>> > >> > > > > According to my understanding, in the clustering
> scenario,
> >>>> the
> >>>> > >> master
> >>>> > >> > > > node
> >>>> > >> > > > > should perform all the deployment actions and the slave
> >>>> nodes
> >>>> > also
> >>>> > >> > need
> >>>> > >> > > > to
> >>>> > >> > > > > perform some deployment actions. For example, the slave
> >>>> nodes
> >>>> > also
> >>>> > >> > > should
> >>>> > >> > > > > handle the process ACTIVATED event so that the process
> >>>> > >> configuration
> >>>> > >> > is
> >>>> > >> > > > > added to the engine and necessary web services are
> created
> >>>> so
> >>>> > that
> >>>> > >> > when
> >>>> > >> > > > the
> >>>> > >> > > > > load balancer send requests to any node in the cluster,
> it
> >>>> is
> >>>> > >> ready
> >>>> > >> > to
> >>>> > >> > > > > accept those requests.
> >>>> > >> > > > >
> >>>> > >> > > > > Regards
> >>>> > >> > > > > Nandika
> >>>> > >> > > > >
> >>>> > >> > > > >
> >>>> > >> > > > >
> >>>> > >> > > > >
> >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> >>>> > >> sathwik.bp@gmail.com>
> >>>> > >> > > > > wrote:
> >>>> > >> > > > >
> >>>> > >> > > > > > Sudharma,
> >>>> > >> > > > > >
> >>>> > >> > > > > > Where are you going to configure the master-slaves, is
> >>>> it in
> >>>> > the
> >>>> > >> > web
> >>>> > >> > > > > > application level or at the load balancer?
> >>>> > >> > > > > >
> >>>> > >> > > > > > regards,
> >>>> > >> > > > > > sathwik
> >>>> > >> > > > > >
> >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> >>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
> >>>> > >> > > > > > wrote:
> >>>> > >> > > > > >
> >>>> > >> > > > > > > Hi Tammo,
> >>>> > >> > > > > > >
> >>>> > >> > > > > > > Can you suggest the best method from these to
> >>>> implement? As
> >>>> > >> > first I
> >>>> > >> > > > > > > suggested the master-slaves scenario I think it is
> >>>> easy to
> >>>> > >> > > implement
> >>>> > >> > > > > than
> >>>> > >> > > > > > > distributed lock scenario. However if you can suggest
> >>>> one
> >>>> > from
> >>>> > >> > > these
> >>>> > >> > > > > two,
> >>>> > >> > > > > > > then I can think about it.
> >>>> > >> > > > > > >
> >>>> > >> > > > > > > Thank you
> >>>> > >> > > > > > >
> >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
> >>>> sathwik.bp@gmail.com>
> >>>> > >> > wrote:
> >>>> > >> > > > > > >
> >>>> > >> > > > > > > > With respect to the hotdeployment,
> >>>> > >> > > > > > > >
> >>>> > >> > > > > > > > We can drop the deployment archive onto the
> >>>> deployment
> >>>> > >> folder.
> >>>> > >> > > > Since
> >>>> > >> > > > > > the
> >>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed lock
> >>>> for
> >>>> > the
> >>>> > >> > > > > > > DeploymentUnit,
> >>>> > >> > > > > > > > only one of the nodes will get the lock and
> initiate
> >>>> the
> >>>> > >> > > > deployment.
> >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
> >>>> acquiring
> >>>> > the
> >>>> > >> > lock
> >>>> > >> > > > and
> >>>> > >> > > > > > > hence
> >>>> > >> > > > > > > > will silently ignore it.
> >>>> > >> > > > > > > >
> >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> >>>> > >> > > > sathwik.bp@gmail.com>
> >>>> > >> > > > > > > > wrote:
> >>>> > >> > > > > > > >
> >>>> > >> > > > > > > > > Hi Tammo,
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > The distributed lock acquisition on the
> >>>> DeploymentUnit
> >>>> > >> should
> >>>> > >> > > be
> >>>> > >> > > > > > added
> >>>> > >> > > > > > > to
> >>>> > >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > When a deployment operation is initiated through
> >>>> the
> >>>> > >> > > > > > > > DeploymentWebService,
> >>>> > >> > > > > > > > > The load balancer routes it to any of the
> available
> >>>> > nodes.
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
> >>>> acquires
> >>>> > the
> >>>> > >> > > > > Distributed
> >>>> > >> > > > > > > > > lock. On the remaining nodes the DeploymentPoller
> >>>> will
> >>>> > >> try to
> >>>> > >> > > > > acquire
> >>>> > >> > > > > > > the
> >>>> > >> > > > > > > > > distributed lock and will not get it and hence
> will
> >>>> > >> silently
> >>>> > >> > > > ignore
> >>>> > >> > > > > > it.
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > Once the routed node completes the deployment, it
> >>>> will
> >>>> > >> > release
> >>>> > >> > > > the
> >>>> > >> > > > > > > lock.
> >>>> > >> > > > > > > > > This way we don't have to stall the
> >>>> DeploymentPoller in
> >>>> > >> other
> >>>> > >> > > > > nodes.
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > Does it answer the concerns?
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > Now, if we give the responsibility of identifying
> >>>> the
> >>>> > >> master
> >>>> > >> > > node
> >>>> > >> > > > > to
> >>>> > >> > > > > > > the
> >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
> >>>> balancer
> >>>> > to
> >>>> > >> > > change
> >>>> > >> > > > > > it's
> >>>> > >> > > > > > > > > configuration about the master node?
> >>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
> >>>> > >> > > > > > > > > node1 -master
> >>>> > >> > > > > > > > > node2 - slave
> >>>> > >> > > > > > > > > node3 - slave
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as
> >>>> master
> >>>> > node,
> >>>> > >> > but
> >>>> > >> > > > > > > hazelcast
> >>>> > >> > > > > > > > > might promote Node3 as master node. They are out
> of
> >>>> > sync.
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > Is this argument valid?
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > regards,
> >>>> > >> > > > > > > > > sathwik
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van
> Lessen <
> >>>> > >> > > > > > > tvanlessen@gmail.com>
> >>>> > >> > > > > > > > > wrote:
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > >> Hi Sudharma,
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >> what do you expect from the "other nodes
> >>>> deployment"?
> >>>> > >> > > > Compilation
> >>>> > >> > > > > is
> >>>> > >> > > > > > > not
> >>>> > >> > > > > > > > >> needed since the CBP file is written to the
> >>>> (shared)
> >>>> > FS.
> >>>> > >> > > > > > Registration
> >>>> > >> > > > > > > is
> >>>> > >> > > > > > > > >> also not needed, since it is done via the shared
> >>>> > >> database.
> >>>> > >> > So
> >>>> > >> > > > the
> >>>> > >> > > > > > only
> >>>> > >> > > > > > > > >> thing that might be needed is to tell the engine
> >>>> that
> >>>> > >> there
> >>>> > >> > > is a
> >>>> > >> > > > > new
> >>>> > >> > > > > > > > >> deployment. I'd need to check that. If this is
> >>>> needed,
> >>>> > I
> >>>> > >> > > revert
> >>>> > >> > > > my
> >>>> > >> > > > > > > last
> >>>> > >> > > > > > > > >> statement, then it is perhaps better to just
> send
> >>>> an
> >>>> > >> event
> >>>> > >> > > over
> >>>> > >> > > > > > > > Hazelcast
> >>>> > >> > > > > > > > >> to all nodes that the deployment has changed.
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >> Best,
> >>>> > >> > > > > > > > >>   Tammo
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
> >>>> subasinghe <
> >>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
> >>>> > >> > > > > > > > >> > wrote:
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >> > Hi Tammo,
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >> > The master node writes meta data. But runtime
> >>>> > >> information
> >>>> > >> > > must
> >>>> > >> > > > > be
> >>>> > >> > > > > > > > >> available
> >>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared, all
> >>>> nodes
> >>>> > will
> >>>> > >> > see
> >>>> > >> > > > the
> >>>> > >> > > > > > > > >> > availability of a new process. My idea is for
> >>>> master
> >>>> > >> node
> >>>> > >> > to
> >>>> > >> > > > > write
> >>>> > >> > > > > > > the
> >>>> > >> > > > > > > > >> meta
> >>>> > >> > > > > > > > >> > data and other nodes to just read the meta
> data
> >>>> and
> >>>> > >> load
> >>>> > >> > > > > > process.So
> >>>> > >> > > > > > > we
> >>>> > >> > > > > > > > >> need
> >>>> > >> > > > > > > > >> > a small delay between master node deployment
> and
> >>>> > other
> >>>> > >> > nodes
> >>>> > >> > > > > > > > deployment.
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >> > Is there anyway to set the delay between
> master
> >>>> node
> >>>> > >> and
> >>>> > >> > > > slaves
> >>>> > >> > > > > > > until
> >>>> > >> > > > > > > > >> > master node finish the deployment?
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >> > Thank you
> >>>> > >> > > > > > > > >> > Sudharma
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> >>>> > >> > > > tvanlessen@gmail.com
> >>>> > >> > > > > >
> >>>> > >> > > > > > > > wrote:
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >> > > Hi Sathwik,
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B
> P <
> >>>> > >> > > > > > > sathwik.bp@gmail.com>
> >>>> > >> > > > > > > > >> > wrote:
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
> >>>> > >> > > > > > > > >> > > >
> >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is the
> >>>> master
> >>>> > >> node
> >>>> > >> > in
> >>>> > >> > > > the
> >>>> > >> > > > > > > > cluster?
> >>>> > >> > > > > > > > >> > > >
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > I think the easiest approach is to always
> >>>> elect the
> >>>> > >> > oldest
> >>>> > >> > > > > node
> >>>> > >> > > > > > in
> >>>> > >> > > > > > > > the
> >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast
> can
> >>>> > easily
> >>>> > >> > asked
> >>>> > >> > > > for
> >>>> > >> > > > > > > this
> >>>> > >> > > > > > > > >> > > information.
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment
> >>>> Pollers
> >>>> > in
> >>>> > >> > the
> >>>> > >> > > > > slave
> >>>> > >> > > > > > > > nodes?
> >>>> > >> > > > > > > > >> > > >
> >>>> > >> > > > > > > > >> > > >
> >>>> > >> > > > > > > > >> > > Absolutely.
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > Suggestion:
> >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
> >>>> Master-SLaves.
> >>>> > Why
> >>>> > >> > not
> >>>> > >> > > > give
> >>>> > >> > > > > > > every
> >>>> > >> > > > > > > > >> node
> >>>> > >> > > > > > > > >> > > in
> >>>> > >> > > > > > > > >> > > > the cluster the same status
> (Active-Active).
> >>>> > >> > > > > > > > >> > > >
> >>>> > >> > > > > > > > >> > > > When a new deployment is made, the load
> >>>> balancer
> >>>> > >> can
> >>>> > >> > > push
> >>>> > >> > > > it
> >>>> > >> > > > > > to
> >>>> > >> > > > > > > > any
> >>>> > >> > > > > > > > >> of
> >>>> > >> > > > > > > > >> > > the
> >>>> > >> > > > > > > > >> > > > available nodes. That node will probably
> >>>> acquire
> >>>> > a
> >>>> > >> > > > > distributed
> >>>> > >> > > > > > > > lock
> >>>> > >> > > > > > > > >> on
> >>>> > >> > > > > > > > >> > > the
> >>>> > >> > > > > > > > >> > > > deployment unit and acts as master for
> that
> >>>> > >> > deployment.
> >>>> > >> > > > This
> >>>> > >> > > > > > > > ensures
> >>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
> >>>> Probably no
> >>>> > >> static
> >>>> > >> > > > > > > > >> configuration of
> >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor in
> the
> >>>> > >> > hazelcast.
> >>>> > >> > > > > > > > >> > > >
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > But this would not allow to have the
> >>>> hotdeployment
> >>>> > >> via
> >>>> > >> > > > > > filesystem
> >>>> > >> > > > > > > > >> still
> >>>> > >> > > > > > > > >> > > enabled, right?
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > Best,
> >>>> > >> > > > > > > > >> > >   Tammo
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> > > --
> >>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> >>>> > >> > > > > > > > >> > >
> >>>> > >> > > > > > > > >> >
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >> --
> >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> >>>> > >> > > > > > > > >>
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > > >
> >>>> > >> > > > > > > >
> >>>> > >> > > > > > >
> >>>> > >> > > > > >
> >>>> > >> > > > >
> >>>> > >> > > >
> >>>> > >> > >
> >>>> > >> >
> >>>> > >>
> >>>> > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Sathwik,

I can't find the attached document. Please kind be enough to resend it.

On 6 June 2015 at 15:42, Sathwik B P <sa...@gmail.com> wrote:

> Refer this one as I have corrected the numbering of steps.
>
> On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com> wrote:
>
>> Hi Sudharma/Tammo,
>>
>> Kindly review attached document on my proposed approach. Let me know if
>> you have any concerns or doubts on it.
>>
>> regards,
>> sathwik
>>
>> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com> wrote:
>>
>>> find response inline
>>>
>>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
>>> suba.11@cse.mrt.ac.lk> wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry for the late reply. I was trying to achieve a proper solution.
>>>> Following is my approach.
>>>>
>>>> 1) I elected master node for deploying purpose. So when a master node
>>>> goes
>>>> down hazelcast will elect the next oldest node for master.
>>>>
>>>
>>>     Perfect
>>>
>>>
>>>> 2) ODEServer can identify whether clustering is enabled or not by
>>>> getting
>>>> the property value in ode-axis2.properties file. So I introduced new
>>>> property called "ode-axis2.hazelcast.clustering.enabled". If there is no
>>>> clustering enabled server will work as it is. If clustering is enabled,
>>>> cluster will be initialized.
>>>>
>>>
>>>     Perfect
>>>
>>>
>>>> 3) In manual deployment its responsibility is taken by the
>>>> DeploymentPoller. So I give the deployment capability to master node,s
>>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to true.So
>>>> others will not be able to go into check() method in DeploymentPoller.
>>>> So
>>>> deployment will be done by only the master node.
>>>>
>>>
>>>     Perfect
>>>
>>>
>>>>
>>>> 4) In DeploymentWebService, I had to consider few cases.If the deploy
>>>> request goes to the master node, it will deploy the process through web
>>>> service.Others pollers will not go into check() method as they are not
>>>> masters. So master can continue without any involvement of others.
>>>>
>>>>
>>>     Perfect
>>>
>>>
>>>> 5) If the deploy request goes to a slave node, it will do up to file
>>>> creation in the file system.Slave will be stopped at that point. As only
>>>> master poller is checking, master can continue from created files in the
>>>> file system.
>>>>
>>>
>>>     DeploymentWebService provides synchronous operations. The status of
>>> the operation should be communicated to the calling client in the same call.
>>>     DeploymentPoller is a backend thread that goes over each and every
>>> directory under the Deployment directory checking for any changes to
>>> deploy.xml in existing processes and deploy newly added processes. This
>>> process is  sequential and time consuming. As the process directories
>>> grows, so does the time taken for execution of the thread.
>>>
>>>     Since the request is on a slave node and the processing is done on
>>> master node, how do you check for the completion of the
>>> deployment/undeployment of processes and respond back to the client since
>>> the web service call is a synchronous operation. As DeploymentPoller is
>>> taking a lot of time in processing, your request will time out right.
>>>
>>>
>>>>
>>>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl. Each
>>>> _deploymentUnits stores only what its server has deployed. So think,
>>>> that a
>>>> master node goes down another master node appears.But its
>>>> __deploymentUnits
>>>> does not have dus which has deployed by the earlier master node. Hence
>>>> it
>>>> will not be able retire earlier version of the process which is
>>>> deployed by
>>>> previous master. So there will two process which are in "ACTIVE" state
>>>
>>>
>>>> 7) To avoid this, I add the ODEServer as an Observer to check when a new
>>>> master is electing, then load all the deployment units out of the
>>>> store. So
>>>> new master node can have all the dus and can retire appropriate version.
>>>> Usually loadAll() is called at the server start-up time. But there is no
>>>> other way to solve this. I tried to use Hazelcast IMap to store all dus
>>>> among all nodes. But it wasn't success as du is not serializable
>>>> object.
>>>>
>>>
>>>> 8) I figured out that we do not need send cluster message to others as
>>>> all
>>>> the dus' data are persisted to the shared DB. So each node can take the
>>>> du
>>>> and retrieve necessary data using already implemented methods in Process
>>>> Store.
>>>>
>>>> 9) But there is an another problem.The axis2 service corresponding to a
>>>> deployed process does not appear on all nodes of the cluster. That is
>>>> because each server add du which is deployed by it to the process
>>>> store.That is why I had use loadAll() when masters are changing. How to
>>>> solve this?
>>>>
>>>
>>>     I do appriciate your efforts in understanding the implmentation and
>>> changes that need to be done. You are bang on it.
>>>     fireEvent(..) is the method that triggers process activation and
>>> necessary service creation.
>>>
>>>
>>>     But with this given apporach from steps 6 to 8, ODE cannot have
>>> atleast 2 Active servers for Load balancing. You are concentrating on only
>>> one active node that will do deployments and cater to process invocations.
>>>     We should also think about scaling ODE to multiple servers to handle
>>> load.
>>>
>>>     What do you think.
>>>
>>>
>>>> Thank you,
>>>> Sudharma
>>>>
>>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
>>>>
>>>> > Sudharma,
>>>> >
>>>> > Any updates?
>>>> >
>>>> > regards,
>>>> > sathwik
>>>> >
>>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Sudharma,
>>>> > >
>>>> > > Can you elaborate on your option 1).
>>>> > >
>>>> > > Response to your option 2).
>>>> > >
>>>> > >     Process Store is the component that handles process metadata,
>>>> > > compilation and deployment in ODE. Integration layers in ODE
>>>> (Axis2, JBI)
>>>> > > use the process store.
>>>> > >     Future implementations of IL for ODE will also use the process
>>>> store.
>>>> > > We should not be thinking of moving the process store functionality
>>>> to
>>>> > the
>>>> > > integration layers.
>>>> > >
>>>> > >
>>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
>>>> > > suba.11@cse.mrt.ac.lk> wrote:
>>>> > >
>>>> > >> Hi,
>>>> > >>
>>>> > >> I understood the problem within dynamic master/slave
>>>> configuration. In
>>>> > my
>>>> > >> approach, when a deployment request is routed to a slave node
>>>> there will
>>>> > >> not be a deployment. I suggest two options to avoid it.
>>>> > >> 1) Have static master/slave configuration only for deploy process
>>>> > >>
>>>> > > 2) Modify the deployment web service to complie and verify the
>>>> process
>>>> > and
>>>> > >> then copy it to the deploy folder irrespective of whether its a
>>>> master
>>>> > or
>>>> > >> slave, then deployment poller should take care of the deployment
>>>> > >>
>>>> > >>
>>>> > >
>>>> > >>
>>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
>>>> > >>
>>>> > >> > Sudharma,
>>>> > >> >
>>>> > >> > We definitely need a master/slave in the hazelcast cluster. This
>>>> is
>>>> > >> > probably needed for the job migration in the Scheduler to
>>>> migrate the
>>>> > >> jobs
>>>> > >> > associated with a down node. Let hold on this topic for future
>>>> > >> discussion.
>>>> > >> >
>>>> > >> > Going by the explanation where the master/slave nodes have
>>>> certain
>>>> > >> > predefined tasks to perform is perfectly fine.
>>>> > >> >
>>>> > >> > I have this scenario,
>>>> > >> >
>>>> > >> > I am using HAProxy as my load balancer and configured 3 nodes in
>>>> the
>>>> > >> > cluster.
>>>> > >> >
>>>> > >> > Node1 - Active
>>>> > >> > Node2 - Active
>>>> > >> > Node3 - Backup
>>>> > >> >
>>>> > >> > Load balancing algorithm: RoundRobin
>>>> > >> >
>>>> > >> > A Backup node (Node3) is one which the load balancer will not
>>>> route
>>>> > >> > requests to, until one of the Active node i.e either Node1 or
>>>> Node2
>>>> > has
>>>> > >> > gone down.
>>>> > >> >
>>>> > >> > All these 3 nodes are also part of the hazelcast cluster as well.
>>>> > >> >
>>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
>>>> leader/master
>>>> > >> and
>>>> > >> > Node2,Node3 as slaves.
>>>> > >> >
>>>> > >> > I initiate the deploy operation on the DeploymentWebService
>>>> which the
>>>> > >> load
>>>> > >> > balancer routes it to one of the Active nodes in the cluster,
>>>> lets say
>>>> > >> it's
>>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
>>>> cluster,
>>>> > >> > deployment is a success.
>>>> > >> >
>>>> > >> > I initiate another deploy operation on the DeploymentWebService
>>>> which
>>>> > >> the
>>>> > >> > load balancer routes it to the next active node which is Node2.
>>>> Since
>>>> > >> Node2
>>>> > >> > is a slave in the Hazelcast cluster, What happens to the
>>>> deployment?
>>>> > >> >
>>>> > >> > regards,
>>>> > >> > sathwik
>>>> > >> >
>>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>>>> > >> > suba.11@cse.mrt.ac.lk
>>>> > >> > > wrote:
>>>> > >> >
>>>> > >> > > Hi,
>>>> > >> > >
>>>> > >> > > I will explain my approach as much as possible. The oldest
>>>> node in
>>>> > the
>>>> > >> > > hazelcast cluster is elected as the master node. In the
>>>> failure of
>>>> > the
>>>> > >> > > master node, next oldest node will be elected as the master
>>>> node.
>>>> > This
>>>> > >> > > master-slave configuration is just for deployment. When the
>>>> > hazelcast
>>>> > >> > > cluster elected the master node, that node becomes a master
>>>> node for
>>>> > >> > > deploying process. So it will do the deploying artifacts. If
>>>> you
>>>> > want
>>>> > >> to
>>>> > >> > > get the idea of electing master node please refer the code
>>>> which I
>>>> > >> have
>>>> > >> > > located in the github. (
>>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>>>> > >> > >
>>>> > >> > > I identified separated actions which should be followed by the
>>>> > master
>>>> > >> and
>>>> > >> > > salve nodes.
>>>> > >> > > Actions which are followed by master node only
>>>> > >> > > 1) create deployment unit
>>>> > >> > > 2) set the version nu to deployment unit
>>>> > >> > > 3) compile deployment unit
>>>> > >> > > 4) scan deployment unit
>>>> > >> > > 5) retire previous versions
>>>> > >> > > Master node and slave nodes should create _processes which
>>>> stores
>>>> > >> > > ProcessConfImpl
>>>> > >> > > Only master node will write the version nu to database, create
>>>> > >> .deployed
>>>> > >> > > file
>>>> > >> > >
>>>> > >> > > So there are some actions which should be followed only by
>>>> master
>>>> > node
>>>> > >> > > while other actions should be followed by all the nodes.The
>>>> idea of
>>>> > >> > having
>>>> > >> > > a master node is deploying artifacts and avoid others from
>>>> writing
>>>> > the
>>>> > >> > > version nu to database.
>>>> > >> > > Whether a node is active or passive, all nodes should do the
>>>> > >> > > deployment.Master
>>>> > >> > > and slaves will follow necessary actions as in above.
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >> > >
>>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com>
>>>> wrote:
>>>> > >> > >
>>>> > >> > > > Nandika,
>>>> > >> > > >
>>>> > >> > > > I very well understand what you have put across, but it's
>>>> > secondary
>>>> > >> to
>>>> > >> > me
>>>> > >> > > > now.
>>>> > >> > > >
>>>> > >> > > > Sudharma,
>>>> > >> > > > My primary concern is to understand at a high level the
>>>> deployment
>>>> > >> > > > architecture and how would master-slave configuration fit
>>>> in. Are
>>>> > >> there
>>>> > >> > > any
>>>> > >> > > > restrictions imposed by the in-progress design?
>>>> > >> > > >
>>>> > >> > > > Firstly, how would ODE process deployment work under these
>>>> cluster
>>>> > >> > > > configurations?
>>>> > >> > > >
>>>> > >> > > > Sample Cluster configurations: A load balancer is
>>>> frontending the
>>>> > >> > > servers.
>>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
>>>> Active or
>>>> > >> > > Passive.
>>>> > >> > > >
>>>> > >> > > > regards,
>>>> > >> > > > sathwik
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > >
>>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>>>> > >> > jayawark@gmail.com
>>>> > >> > > >
>>>> > >> > > > wrote:
>>>> > >> > > >
>>>> > >> > > > > Hi Sathwik,
>>>> > >> > > > >
>>>> > >> > > > > According to my understanding, in the clustering scenario,
>>>> the
>>>> > >> master
>>>> > >> > > > node
>>>> > >> > > > > should perform all the deployment actions and the slave
>>>> nodes
>>>> > also
>>>> > >> > need
>>>> > >> > > > to
>>>> > >> > > > > perform some deployment actions. For example, the slave
>>>> nodes
>>>> > also
>>>> > >> > > should
>>>> > >> > > > > handle the process ACTIVATED event so that the process
>>>> > >> configuration
>>>> > >> > is
>>>> > >> > > > > added to the engine and necessary web services are created
>>>> so
>>>> > that
>>>> > >> > when
>>>> > >> > > > the
>>>> > >> > > > > load balancer send requests to any node in the cluster, it
>>>> is
>>>> > >> ready
>>>> > >> > to
>>>> > >> > > > > accept those requests.
>>>> > >> > > > >
>>>> > >> > > > > Regards
>>>> > >> > > > > Nandika
>>>> > >> > > > >
>>>> > >> > > > >
>>>> > >> > > > >
>>>> > >> > > > >
>>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>>>> > >> sathwik.bp@gmail.com>
>>>> > >> > > > > wrote:
>>>> > >> > > > >
>>>> > >> > > > > > Sudharma,
>>>> > >> > > > > >
>>>> > >> > > > > > Where are you going to configure the master-slaves, is
>>>> it in
>>>> > the
>>>> > >> > web
>>>> > >> > > > > > application level or at the load balancer?
>>>> > >> > > > > >
>>>> > >> > > > > > regards,
>>>> > >> > > > > > sathwik
>>>> > >> > > > > >
>>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
>>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
>>>> > >> > > > > > wrote:
>>>> > >> > > > > >
>>>> > >> > > > > > > Hi Tammo,
>>>> > >> > > > > > >
>>>> > >> > > > > > > Can you suggest the best method from these to
>>>> implement? As
>>>> > >> > first I
>>>> > >> > > > > > > suggested the master-slaves scenario I think it is
>>>> easy to
>>>> > >> > > implement
>>>> > >> > > > > than
>>>> > >> > > > > > > distributed lock scenario. However if you can suggest
>>>> one
>>>> > from
>>>> > >> > > these
>>>> > >> > > > > two,
>>>> > >> > > > > > > then I can think about it.
>>>> > >> > > > > > >
>>>> > >> > > > > > > Thank you
>>>> > >> > > > > > >
>>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
>>>> sathwik.bp@gmail.com>
>>>> > >> > wrote:
>>>> > >> > > > > > >
>>>> > >> > > > > > > > With respect to the hotdeployment,
>>>> > >> > > > > > > >
>>>> > >> > > > > > > > We can drop the deployment archive onto the
>>>> deployment
>>>> > >> folder.
>>>> > >> > > > Since
>>>> > >> > > > > > the
>>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed lock
>>>> for
>>>> > the
>>>> > >> > > > > > > DeploymentUnit,
>>>> > >> > > > > > > > only one of the nodes will get the lock and initiate
>>>> the
>>>> > >> > > > deployment.
>>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
>>>> acquiring
>>>> > the
>>>> > >> > lock
>>>> > >> > > > and
>>>> > >> > > > > > > hence
>>>> > >> > > > > > > > will silently ignore it.
>>>> > >> > > > > > > >
>>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>>>> > >> > > > sathwik.bp@gmail.com>
>>>> > >> > > > > > > > wrote:
>>>> > >> > > > > > > >
>>>> > >> > > > > > > > > Hi Tammo,
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > The distributed lock acquisition on the
>>>> DeploymentUnit
>>>> > >> should
>>>> > >> > > be
>>>> > >> > > > > > added
>>>> > >> > > > > > > to
>>>> > >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > When a deployment operation is initiated through
>>>> the
>>>> > >> > > > > > > > DeploymentWebService,
>>>> > >> > > > > > > > > The load balancer routes it to any of the available
>>>> > nodes.
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
>>>> acquires
>>>> > the
>>>> > >> > > > > Distributed
>>>> > >> > > > > > > > > lock. On the remaining nodes the DeploymentPoller
>>>> will
>>>> > >> try to
>>>> > >> > > > > acquire
>>>> > >> > > > > > > the
>>>> > >> > > > > > > > > distributed lock and will not get it and hence will
>>>> > >> silently
>>>> > >> > > > ignore
>>>> > >> > > > > > it.
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > Once the routed node completes the deployment, it
>>>> will
>>>> > >> > release
>>>> > >> > > > the
>>>> > >> > > > > > > lock.
>>>> > >> > > > > > > > > This way we don't have to stall the
>>>> DeploymentPoller in
>>>> > >> other
>>>> > >> > > > > nodes.
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > Does it answer the concerns?
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > Now, if we give the responsibility of identifying
>>>> the
>>>> > >> master
>>>> > >> > > node
>>>> > >> > > > > to
>>>> > >> > > > > > > the
>>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
>>>> balancer
>>>> > to
>>>> > >> > > change
>>>> > >> > > > > > it's
>>>> > >> > > > > > > > > configuration about the master node?
>>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
>>>> > >> > > > > > > > > node1 -master
>>>> > >> > > > > > > > > node2 - slave
>>>> > >> > > > > > > > > node3 - slave
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as
>>>> master
>>>> > node,
>>>> > >> > but
>>>> > >> > > > > > > hazelcast
>>>> > >> > > > > > > > > might promote Node3 as master node. They are out of
>>>> > sync.
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > Is this argument valid?
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > regards,
>>>> > >> > > > > > > > > sathwik
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
>>>> > >> > > > > > > tvanlessen@gmail.com>
>>>> > >> > > > > > > > > wrote:
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > >> Hi Sudharma,
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >> what do you expect from the "other nodes
>>>> deployment"?
>>>> > >> > > > Compilation
>>>> > >> > > > > is
>>>> > >> > > > > > > not
>>>> > >> > > > > > > > >> needed since the CBP file is written to the
>>>> (shared)
>>>> > FS.
>>>> > >> > > > > > Registration
>>>> > >> > > > > > > is
>>>> > >> > > > > > > > >> also not needed, since it is done via the shared
>>>> > >> database.
>>>> > >> > So
>>>> > >> > > > the
>>>> > >> > > > > > only
>>>> > >> > > > > > > > >> thing that might be needed is to tell the engine
>>>> that
>>>> > >> there
>>>> > >> > > is a
>>>> > >> > > > > new
>>>> > >> > > > > > > > >> deployment. I'd need to check that. If this is
>>>> needed,
>>>> > I
>>>> > >> > > revert
>>>> > >> > > > my
>>>> > >> > > > > > > last
>>>> > >> > > > > > > > >> statement, then it is perhaps better to just send
>>>> an
>>>> > >> event
>>>> > >> > > over
>>>> > >> > > > > > > > Hazelcast
>>>> > >> > > > > > > > >> to all nodes that the deployment has changed.
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >> Best,
>>>> > >> > > > > > > > >>   Tammo
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
>>>> subasinghe <
>>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
>>>> > >> > > > > > > > >> > wrote:
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >> > Hi Tammo,
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >> > The master node writes meta data. But runtime
>>>> > >> information
>>>> > >> > > must
>>>> > >> > > > > be
>>>> > >> > > > > > > > >> available
>>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared, all
>>>> nodes
>>>> > will
>>>> > >> > see
>>>> > >> > > > the
>>>> > >> > > > > > > > >> > availability of a new process. My idea is for
>>>> master
>>>> > >> node
>>>> > >> > to
>>>> > >> > > > > write
>>>> > >> > > > > > > the
>>>> > >> > > > > > > > >> meta
>>>> > >> > > > > > > > >> > data and other nodes to just read the meta data
>>>> and
>>>> > >> load
>>>> > >> > > > > > process.So
>>>> > >> > > > > > > we
>>>> > >> > > > > > > > >> need
>>>> > >> > > > > > > > >> > a small delay between master node deployment and
>>>> > other
>>>> > >> > nodes
>>>> > >> > > > > > > > deployment.
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >> > Is there anyway to set the delay between master
>>>> node
>>>> > >> and
>>>> > >> > > > slaves
>>>> > >> > > > > > > until
>>>> > >> > > > > > > > >> > master node finish the deployment?
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >> > Thank you
>>>> > >> > > > > > > > >> > Sudharma
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
>>>> > >> > > > tvanlessen@gmail.com
>>>> > >> > > > > >
>>>> > >> > > > > > > > wrote:
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >> > > Hi Sathwik,
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
>>>> > >> > > > > > > sathwik.bp@gmail.com>
>>>> > >> > > > > > > > >> > wrote:
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
>>>> > >> > > > > > > > >> > > >
>>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is the
>>>> master
>>>> > >> node
>>>> > >> > in
>>>> > >> > > > the
>>>> > >> > > > > > > > cluster?
>>>> > >> > > > > > > > >> > > >
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > I think the easiest approach is to always
>>>> elect the
>>>> > >> > oldest
>>>> > >> > > > > node
>>>> > >> > > > > > in
>>>> > >> > > > > > > > the
>>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can
>>>> > easily
>>>> > >> > asked
>>>> > >> > > > for
>>>> > >> > > > > > > this
>>>> > >> > > > > > > > >> > > information.
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment
>>>> Pollers
>>>> > in
>>>> > >> > the
>>>> > >> > > > > slave
>>>> > >> > > > > > > > nodes?
>>>> > >> > > > > > > > >> > > >
>>>> > >> > > > > > > > >> > > >
>>>> > >> > > > > > > > >> > > Absolutely.
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > Suggestion:
>>>> > >> > > > > > > > >> > > > I am not sure whether do we need
>>>> Master-SLaves.
>>>> > Why
>>>> > >> > not
>>>> > >> > > > give
>>>> > >> > > > > > > every
>>>> > >> > > > > > > > >> node
>>>> > >> > > > > > > > >> > > in
>>>> > >> > > > > > > > >> > > > the cluster the same status (Active-Active).
>>>> > >> > > > > > > > >> > > >
>>>> > >> > > > > > > > >> > > > When a new deployment is made, the load
>>>> balancer
>>>> > >> can
>>>> > >> > > push
>>>> > >> > > > it
>>>> > >> > > > > > to
>>>> > >> > > > > > > > any
>>>> > >> > > > > > > > >> of
>>>> > >> > > > > > > > >> > > the
>>>> > >> > > > > > > > >> > > > available nodes. That node will probably
>>>> acquire
>>>> > a
>>>> > >> > > > > distributed
>>>> > >> > > > > > > > lock
>>>> > >> > > > > > > > >> on
>>>> > >> > > > > > > > >> > > the
>>>> > >> > > > > > > > >> > > > deployment unit and acts as master for that
>>>> > >> > deployment.
>>>> > >> > > > This
>>>> > >> > > > > > > > ensures
>>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
>>>> Probably no
>>>> > >> static
>>>> > >> > > > > > > > >> configuration of
>>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
>>>> > >> > hazelcast.
>>>> > >> > > > > > > > >> > > >
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > But this would not allow to have the
>>>> hotdeployment
>>>> > >> via
>>>> > >> > > > > > filesystem
>>>> > >> > > > > > > > >> still
>>>> > >> > > > > > > > >> > > enabled, right?
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > Best,
>>>> > >> > > > > > > > >> > >   Tammo
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> > > --
>>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>>>> > >> > > > > > > > >> > >
>>>> > >> > > > > > > > >> >
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >> --
>>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>>>> > >> > > > > > > > >>
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > > >
>>>> > >> > > > > > > >
>>>> > >> > > > > > >
>>>> > >> > > > > >
>>>> > >> > > > >
>>>> > >> > > >
>>>> > >> > >
>>>> > >> >
>>>> > >>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Refer this one as I have corrected the numbering of steps.

On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sa...@gmail.com> wrote:

> Hi Sudharma/Tammo,
>
> Kindly review attached document on my proposed approach. Let me know if
> you have any concerns or doubts on it.
>
> regards,
> sathwik
>
> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com> wrote:
>
>> find response inline
>>
>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
>> suba.11@cse.mrt.ac.lk> wrote:
>>
>>> Hi,
>>>
>>> Sorry for the late reply. I was trying to achieve a proper solution.
>>> Following is my approach.
>>>
>>> 1) I elected master node for deploying purpose. So when a master node
>>> goes
>>> down hazelcast will elect the next oldest node for master.
>>>
>>
>>     Perfect
>>
>>
>>> 2) ODEServer can identify whether clustering is enabled or not by getting
>>> the property value in ode-axis2.properties file. So I introduced new
>>> property called "ode-axis2.hazelcast.clustering.enabled". If there is no
>>> clustering enabled server will work as it is. If clustering is enabled,
>>> cluster will be initialized.
>>>
>>
>>     Perfect
>>
>>
>>> 3) In manual deployment its responsibility is taken by the
>>> DeploymentPoller. So I give the deployment capability to master node,s
>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to true.So
>>> others will not be able to go into check() method in DeploymentPoller. So
>>> deployment will be done by only the master node.
>>>
>>
>>     Perfect
>>
>>
>>>
>>> 4) In DeploymentWebService, I had to consider few cases.If the deploy
>>> request goes to the master node, it will deploy the process through web
>>> service.Others pollers will not go into check() method as they are not
>>> masters. So master can continue without any involvement of others.
>>>
>>>
>>     Perfect
>>
>>
>>> 5) If the deploy request goes to a slave node, it will do up to file
>>> creation in the file system.Slave will be stopped at that point. As only
>>> master poller is checking, master can continue from created files in the
>>> file system.
>>>
>>
>>     DeploymentWebService provides synchronous operations. The status of
>> the operation should be communicated to the calling client in the same call.
>>     DeploymentPoller is a backend thread that goes over each and every
>> directory under the Deployment directory checking for any changes to
>> deploy.xml in existing processes and deploy newly added processes. This
>> process is  sequential and time consuming. As the process directories
>> grows, so does the time taken for execution of the thread.
>>
>>     Since the request is on a slave node and the processing is done on
>> master node, how do you check for the completion of the
>> deployment/undeployment of processes and respond back to the client since
>> the web service call is a synchronous operation. As DeploymentPoller is
>> taking a lot of time in processing, your request will time out right.
>>
>>
>>>
>>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl. Each
>>> _deploymentUnits stores only what its server has deployed. So think,
>>> that a
>>> master node goes down another master node appears.But its
>>> __deploymentUnits
>>> does not have dus which has deployed by the earlier master node. Hence it
>>> will not be able retire earlier version of the process which is deployed
>>> by
>>> previous master. So there will two process which are in "ACTIVE" state
>>
>>
>>> 7) To avoid this, I add the ODEServer as an Observer to check when a new
>>> master is electing, then load all the deployment units out of the store.
>>> So
>>> new master node can have all the dus and can retire appropriate version.
>>> Usually loadAll() is called at the server start-up time. But there is no
>>> other way to solve this. I tried to use Hazelcast IMap to store all dus
>>> among all nodes. But it wasn't success as du is not serializable object.
>>>
>>
>>> 8) I figured out that we do not need send cluster message to others as
>>> all
>>> the dus' data are persisted to the shared DB. So each node can take the
>>> du
>>> and retrieve necessary data using already implemented methods in Process
>>> Store.
>>>
>>> 9) But there is an another problem.The axis2 service corresponding to a
>>> deployed process does not appear on all nodes of the cluster. That is
>>> because each server add du which is deployed by it to the process
>>> store.That is why I had use loadAll() when masters are changing. How to
>>> solve this?
>>>
>>
>>     I do appriciate your efforts in understanding the implmentation and
>> changes that need to be done. You are bang on it.
>>     fireEvent(..) is the method that triggers process activation and
>> necessary service creation.
>>
>>
>>     But with this given apporach from steps 6 to 8, ODE cannot have
>> atleast 2 Active servers for Load balancing. You are concentrating on only
>> one active node that will do deployments and cater to process invocations.
>>     We should also think about scaling ODE to multiple servers to handle
>> load.
>>
>>     What do you think.
>>
>>
>>> Thank you,
>>> Sudharma
>>>
>>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
>>>
>>> > Sudharma,
>>> >
>>> > Any updates?
>>> >
>>> > regards,
>>> > sathwik
>>> >
>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com>
>>> wrote:
>>> >
>>> > > Sudharma,
>>> > >
>>> > > Can you elaborate on your option 1).
>>> > >
>>> > > Response to your option 2).
>>> > >
>>> > >     Process Store is the component that handles process metadata,
>>> > > compilation and deployment in ODE. Integration layers in ODE (Axis2,
>>> JBI)
>>> > > use the process store.
>>> > >     Future implementations of IL for ODE will also use the process
>>> store.
>>> > > We should not be thinking of moving the process store functionality
>>> to
>>> > the
>>> > > integration layers.
>>> > >
>>> > >
>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
>>> > > suba.11@cse.mrt.ac.lk> wrote:
>>> > >
>>> > >> Hi,
>>> > >>
>>> > >> I understood the problem within dynamic master/slave configuration.
>>> In
>>> > my
>>> > >> approach, when a deployment request is routed to a slave node there
>>> will
>>> > >> not be a deployment. I suggest two options to avoid it.
>>> > >> 1) Have static master/slave configuration only for deploy process
>>> > >>
>>> > > 2) Modify the deployment web service to complie and verify the
>>> process
>>> > and
>>> > >> then copy it to the deploy folder irrespective of whether its a
>>> master
>>> > or
>>> > >> slave, then deployment poller should take care of the deployment
>>> > >>
>>> > >>
>>> > >
>>> > >>
>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
>>> > >>
>>> > >> > Sudharma,
>>> > >> >
>>> > >> > We definitely need a master/slave in the hazelcast cluster. This
>>> is
>>> > >> > probably needed for the job migration in the Scheduler to migrate
>>> the
>>> > >> jobs
>>> > >> > associated with a down node. Let hold on this topic for future
>>> > >> discussion.
>>> > >> >
>>> > >> > Going by the explanation where the master/slave nodes have certain
>>> > >> > predefined tasks to perform is perfectly fine.
>>> > >> >
>>> > >> > I have this scenario,
>>> > >> >
>>> > >> > I am using HAProxy as my load balancer and configured 3 nodes in
>>> the
>>> > >> > cluster.
>>> > >> >
>>> > >> > Node1 - Active
>>> > >> > Node2 - Active
>>> > >> > Node3 - Backup
>>> > >> >
>>> > >> > Load balancing algorithm: RoundRobin
>>> > >> >
>>> > >> > A Backup node (Node3) is one which the load balancer will not
>>> route
>>> > >> > requests to, until one of the Active node i.e either Node1 or
>>> Node2
>>> > has
>>> > >> > gone down.
>>> > >> >
>>> > >> > All these 3 nodes are also part of the hazelcast cluster as well.
>>> > >> >
>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
>>> leader/master
>>> > >> and
>>> > >> > Node2,Node3 as slaves.
>>> > >> >
>>> > >> > I initiate the deploy operation on the DeploymentWebService which
>>> the
>>> > >> load
>>> > >> > balancer routes it to one of the Active nodes in the cluster,
>>> lets say
>>> > >> it's
>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
>>> cluster,
>>> > >> > deployment is a success.
>>> > >> >
>>> > >> > I initiate another deploy operation on the DeploymentWebService
>>> which
>>> > >> the
>>> > >> > load balancer routes it to the next active node which is Node2.
>>> Since
>>> > >> Node2
>>> > >> > is a slave in the Hazelcast cluster, What happens to the
>>> deployment?
>>> > >> >
>>> > >> > regards,
>>> > >> > sathwik
>>> > >> >
>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>>> > >> > suba.11@cse.mrt.ac.lk
>>> > >> > > wrote:
>>> > >> >
>>> > >> > > Hi,
>>> > >> > >
>>> > >> > > I will explain my approach as much as possible. The oldest node
>>> in
>>> > the
>>> > >> > > hazelcast cluster is elected as the master node. In the failure
>>> of
>>> > the
>>> > >> > > master node, next oldest node will be elected as the master
>>> node.
>>> > This
>>> > >> > > master-slave configuration is just for deployment. When the
>>> > hazelcast
>>> > >> > > cluster elected the master node, that node becomes a master
>>> node for
>>> > >> > > deploying process. So it will do the deploying artifacts. If you
>>> > want
>>> > >> to
>>> > >> > > get the idea of electing master node please refer the code
>>> which I
>>> > >> have
>>> > >> > > located in the github. (
>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>>> > >> > >
>>> > >> > > I identified separated actions which should be followed by the
>>> > master
>>> > >> and
>>> > >> > > salve nodes.
>>> > >> > > Actions which are followed by master node only
>>> > >> > > 1) create deployment unit
>>> > >> > > 2) set the version nu to deployment unit
>>> > >> > > 3) compile deployment unit
>>> > >> > > 4) scan deployment unit
>>> > >> > > 5) retire previous versions
>>> > >> > > Master node and slave nodes should create _processes which
>>> stores
>>> > >> > > ProcessConfImpl
>>> > >> > > Only master node will write the version nu to database, create
>>> > >> .deployed
>>> > >> > > file
>>> > >> > >
>>> > >> > > So there are some actions which should be followed only by
>>> master
>>> > node
>>> > >> > > while other actions should be followed by all the nodes.The
>>> idea of
>>> > >> > having
>>> > >> > > a master node is deploying artifacts and avoid others from
>>> writing
>>> > the
>>> > >> > > version nu to database.
>>> > >> > > Whether a node is active or passive, all nodes should do the
>>> > >> > > deployment.Master
>>> > >> > > and slaves will follow necessary actions as in above.
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com>
>>> wrote:
>>> > >> > >
>>> > >> > > > Nandika,
>>> > >> > > >
>>> > >> > > > I very well understand what you have put across, but it's
>>> > secondary
>>> > >> to
>>> > >> > me
>>> > >> > > > now.
>>> > >> > > >
>>> > >> > > > Sudharma,
>>> > >> > > > My primary concern is to understand at a high level the
>>> deployment
>>> > >> > > > architecture and how would master-slave configuration fit in.
>>> Are
>>> > >> there
>>> > >> > > any
>>> > >> > > > restrictions imposed by the in-progress design?
>>> > >> > > >
>>> > >> > > > Firstly, how would ODE process deployment work under these
>>> cluster
>>> > >> > > > configurations?
>>> > >> > > >
>>> > >> > > > Sample Cluster configurations: A load balancer is frontending
>>> the
>>> > >> > > servers.
>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
>>> Active or
>>> > >> > > Passive.
>>> > >> > > >
>>> > >> > > > regards,
>>> > >> > > > sathwik
>>> > >> > > >
>>> > >> > > >
>>> > >> > > >
>>> > >> > > >
>>> > >> > > >
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>>> > >> > jayawark@gmail.com
>>> > >> > > >
>>> > >> > > > wrote:
>>> > >> > > >
>>> > >> > > > > Hi Sathwik,
>>> > >> > > > >
>>> > >> > > > > According to my understanding, in the clustering scenario,
>>> the
>>> > >> master
>>> > >> > > > node
>>> > >> > > > > should perform all the deployment actions and the slave
>>> nodes
>>> > also
>>> > >> > need
>>> > >> > > > to
>>> > >> > > > > perform some deployment actions. For example, the slave
>>> nodes
>>> > also
>>> > >> > > should
>>> > >> > > > > handle the process ACTIVATED event so that the process
>>> > >> configuration
>>> > >> > is
>>> > >> > > > > added to the engine and necessary web services are created
>>> so
>>> > that
>>> > >> > when
>>> > >> > > > the
>>> > >> > > > > load balancer send requests to any node in the cluster, it
>>> is
>>> > >> ready
>>> > >> > to
>>> > >> > > > > accept those requests.
>>> > >> > > > >
>>> > >> > > > > Regards
>>> > >> > > > > Nandika
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>>> > >> sathwik.bp@gmail.com>
>>> > >> > > > > wrote:
>>> > >> > > > >
>>> > >> > > > > > Sudharma,
>>> > >> > > > > >
>>> > >> > > > > > Where are you going to configure the master-slaves, is it
>>> in
>>> > the
>>> > >> > web
>>> > >> > > > > > application level or at the load balancer?
>>> > >> > > > > >
>>> > >> > > > > > regards,
>>> > >> > > > > > sathwik
>>> > >> > > > > >
>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
>>> > >> > > > > > suba.11@cse.mrt.ac.lk>
>>> > >> > > > > > wrote:
>>> > >> > > > > >
>>> > >> > > > > > > Hi Tammo,
>>> > >> > > > > > >
>>> > >> > > > > > > Can you suggest the best method from these to
>>> implement? As
>>> > >> > first I
>>> > >> > > > > > > suggested the master-slaves scenario I think it is easy
>>> to
>>> > >> > > implement
>>> > >> > > > > than
>>> > >> > > > > > > distributed lock scenario. However if you can suggest
>>> one
>>> > from
>>> > >> > > these
>>> > >> > > > > two,
>>> > >> > > > > > > then I can think about it.
>>> > >> > > > > > >
>>> > >> > > > > > > Thank you
>>> > >> > > > > > >
>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
>>> sathwik.bp@gmail.com>
>>> > >> > wrote:
>>> > >> > > > > > >
>>> > >> > > > > > > > With respect to the hotdeployment,
>>> > >> > > > > > > >
>>> > >> > > > > > > > We can drop the deployment archive onto the deployment
>>> > >> folder.
>>> > >> > > > Since
>>> > >> > > > > > the
>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed lock
>>> for
>>> > the
>>> > >> > > > > > > DeploymentUnit,
>>> > >> > > > > > > > only one of the nodes will get the lock and initiate
>>> the
>>> > >> > > > deployment.
>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
>>> acquiring
>>> > the
>>> > >> > lock
>>> > >> > > > and
>>> > >> > > > > > > hence
>>> > >> > > > > > > > will silently ignore it.
>>> > >> > > > > > > >
>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>>> > >> > > > sathwik.bp@gmail.com>
>>> > >> > > > > > > > wrote:
>>> > >> > > > > > > >
>>> > >> > > > > > > > > Hi Tammo,
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > The distributed lock acquisition on the
>>> DeploymentUnit
>>> > >> should
>>> > >> > > be
>>> > >> > > > > > added
>>> > >> > > > > > > to
>>> > >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > When a deployment operation is initiated through the
>>> > >> > > > > > > > DeploymentWebService,
>>> > >> > > > > > > > > The load balancer routes it to any of the available
>>> > nodes.
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
>>> acquires
>>> > the
>>> > >> > > > > Distributed
>>> > >> > > > > > > > > lock. On the remaining nodes the DeploymentPoller
>>> will
>>> > >> try to
>>> > >> > > > > acquire
>>> > >> > > > > > > the
>>> > >> > > > > > > > > distributed lock and will not get it and hence will
>>> > >> silently
>>> > >> > > > ignore
>>> > >> > > > > > it.
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > Once the routed node completes the deployment, it
>>> will
>>> > >> > release
>>> > >> > > > the
>>> > >> > > > > > > lock.
>>> > >> > > > > > > > > This way we don't have to stall the
>>> DeploymentPoller in
>>> > >> other
>>> > >> > > > > nodes.
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > Does it answer the concerns?
>>> > >> > > > > > > > >
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > Now, if we give the responsibility of identifying
>>> the
>>> > >> master
>>> > >> > > node
>>> > >> > > > > to
>>> > >> > > > > > > the
>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
>>> balancer
>>> > to
>>> > >> > > change
>>> > >> > > > > > it's
>>> > >> > > > > > > > > configuration about the master node?
>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
>>> > >> > > > > > > > > node1 -master
>>> > >> > > > > > > > > node2 - slave
>>> > >> > > > > > > > > node3 - slave
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as master
>>> > node,
>>> > >> > but
>>> > >> > > > > > > hazelcast
>>> > >> > > > > > > > > might promote Node3 as master node. They are out of
>>> > sync.
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > Is this argument valid?
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > regards,
>>> > >> > > > > > > > > sathwik
>>> > >> > > > > > > > >
>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
>>> > >> > > > > > > tvanlessen@gmail.com>
>>> > >> > > > > > > > > wrote:
>>> > >> > > > > > > > >
>>> > >> > > > > > > > >> Hi Sudharma,
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >> what do you expect from the "other nodes
>>> deployment"?
>>> > >> > > > Compilation
>>> > >> > > > > is
>>> > >> > > > > > > not
>>> > >> > > > > > > > >> needed since the CBP file is written to the
>>> (shared)
>>> > FS.
>>> > >> > > > > > Registration
>>> > >> > > > > > > is
>>> > >> > > > > > > > >> also not needed, since it is done via the shared
>>> > >> database.
>>> > >> > So
>>> > >> > > > the
>>> > >> > > > > > only
>>> > >> > > > > > > > >> thing that might be needed is to tell the engine
>>> that
>>> > >> there
>>> > >> > > is a
>>> > >> > > > > new
>>> > >> > > > > > > > >> deployment. I'd need to check that. If this is
>>> needed,
>>> > I
>>> > >> > > revert
>>> > >> > > > my
>>> > >> > > > > > > last
>>> > >> > > > > > > > >> statement, then it is perhaps better to just send
>>> an
>>> > >> event
>>> > >> > > over
>>> > >> > > > > > > > Hazelcast
>>> > >> > > > > > > > >> to all nodes that the deployment has changed.
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >> Best,
>>> > >> > > > > > > > >>   Tammo
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
>>> subasinghe <
>>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
>>> > >> > > > > > > > >> > wrote:
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >> > Hi Tammo,
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >> > The master node writes meta data. But runtime
>>> > >> information
>>> > >> > > must
>>> > >> > > > > be
>>> > >> > > > > > > > >> available
>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared, all
>>> nodes
>>> > will
>>> > >> > see
>>> > >> > > > the
>>> > >> > > > > > > > >> > availability of a new process. My idea is for
>>> master
>>> > >> node
>>> > >> > to
>>> > >> > > > > write
>>> > >> > > > > > > the
>>> > >> > > > > > > > >> meta
>>> > >> > > > > > > > >> > data and other nodes to just read the meta data
>>> and
>>> > >> load
>>> > >> > > > > > process.So
>>> > >> > > > > > > we
>>> > >> > > > > > > > >> need
>>> > >> > > > > > > > >> > a small delay between master node deployment and
>>> > other
>>> > >> > nodes
>>> > >> > > > > > > > deployment.
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >> > Is there anyway to set the delay between master
>>> node
>>> > >> and
>>> > >> > > > slaves
>>> > >> > > > > > > until
>>> > >> > > > > > > > >> > master node finish the deployment?
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >> > Thank you
>>> > >> > > > > > > > >> > Sudharma
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
>>> > >> > > > tvanlessen@gmail.com
>>> > >> > > > > >
>>> > >> > > > > > > > wrote:
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >> > > Hi Sathwik,
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
>>> > >> > > > > > > sathwik.bp@gmail.com>
>>> > >> > > > > > > > >> > wrote:
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
>>> > >> > > > > > > > >> > > >
>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is the
>>> master
>>> > >> node
>>> > >> > in
>>> > >> > > > the
>>> > >> > > > > > > > cluster?
>>> > >> > > > > > > > >> > > >
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > I think the easiest approach is to always
>>> elect the
>>> > >> > oldest
>>> > >> > > > > node
>>> > >> > > > > > in
>>> > >> > > > > > > > the
>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can
>>> > easily
>>> > >> > asked
>>> > >> > > > for
>>> > >> > > > > > > this
>>> > >> > > > > > > > >> > > information.
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment
>>> Pollers
>>> > in
>>> > >> > the
>>> > >> > > > > slave
>>> > >> > > > > > > > nodes?
>>> > >> > > > > > > > >> > > >
>>> > >> > > > > > > > >> > > >
>>> > >> > > > > > > > >> > > Absolutely.
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > Suggestion:
>>> > >> > > > > > > > >> > > > I am not sure whether do we need
>>> Master-SLaves.
>>> > Why
>>> > >> > not
>>> > >> > > > give
>>> > >> > > > > > > every
>>> > >> > > > > > > > >> node
>>> > >> > > > > > > > >> > > in
>>> > >> > > > > > > > >> > > > the cluster the same status (Active-Active).
>>> > >> > > > > > > > >> > > >
>>> > >> > > > > > > > >> > > > When a new deployment is made, the load
>>> balancer
>>> > >> can
>>> > >> > > push
>>> > >> > > > it
>>> > >> > > > > > to
>>> > >> > > > > > > > any
>>> > >> > > > > > > > >> of
>>> > >> > > > > > > > >> > > the
>>> > >> > > > > > > > >> > > > available nodes. That node will probably
>>> acquire
>>> > a
>>> > >> > > > > distributed
>>> > >> > > > > > > > lock
>>> > >> > > > > > > > >> on
>>> > >> > > > > > > > >> > > the
>>> > >> > > > > > > > >> > > > deployment unit and acts as master for that
>>> > >> > deployment.
>>> > >> > > > This
>>> > >> > > > > > > > ensures
>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes. Probably
>>> no
>>> > >> static
>>> > >> > > > > > > > >> configuration of
>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
>>> > >> > hazelcast.
>>> > >> > > > > > > > >> > > >
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > But this would not allow to have the
>>> hotdeployment
>>> > >> via
>>> > >> > > > > > filesystem
>>> > >> > > > > > > > >> still
>>> > >> > > > > > > > >> > > enabled, right?
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > Best,
>>> > >> > > > > > > > >> > >   Tammo
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> > > --
>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>>> > >> > > > > > > > >> > >
>>> > >> > > > > > > > >> >
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >> --
>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>>> > >> > > > > > > > >>
>>> > >> > > > > > > > >
>>> > >> > > > > > > > >
>>> > >> > > > > > > >
>>> > >> > > > > > >
>>> > >> > > > > >
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Hi Sudharma/Tammo,

Kindly review attached document on my proposed approach. Let me know if you
have any concerns or doubts on it.

regards,
sathwik

On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sa...@gmail.com> wrote:

> find response inline
>
> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk> wrote:
>
>> Hi,
>>
>> Sorry for the late reply. I was trying to achieve a proper solution.
>> Following is my approach.
>>
>> 1) I elected master node for deploying purpose. So when a master node goes
>> down hazelcast will elect the next oldest node for master.
>>
>
>     Perfect
>
>
>> 2) ODEServer can identify whether clustering is enabled or not by getting
>> the property value in ode-axis2.properties file. So I introduced new
>> property called "ode-axis2.hazelcast.clustering.enabled". If there is no
>> clustering enabled server will work as it is. If clustering is enabled,
>> cluster will be initialized.
>>
>
>     Perfect
>
>
>> 3) In manual deployment its responsibility is taken by the
>> DeploymentPoller. So I give the deployment capability to master node,s
>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to true.So
>> others will not be able to go into check() method in DeploymentPoller. So
>> deployment will be done by only the master node.
>>
>
>     Perfect
>
>
>>
>> 4) In DeploymentWebService, I had to consider few cases.If the deploy
>> request goes to the master node, it will deploy the process through web
>> service.Others pollers will not go into check() method as they are not
>> masters. So master can continue without any involvement of others.
>>
>>
>     Perfect
>
>
>> 5) If the deploy request goes to a slave node, it will do up to file
>> creation in the file system.Slave will be stopped at that point. As only
>> master poller is checking, master can continue from created files in the
>> file system.
>>
>
>     DeploymentWebService provides synchronous operations. The status of
> the operation should be communicated to the calling client in the same call.
>     DeploymentPoller is a backend thread that goes over each and every
> directory under the Deployment directory checking for any changes to
> deploy.xml in existing processes and deploy newly added processes. This
> process is  sequential and time consuming. As the process directories
> grows, so does the time taken for execution of the thread.
>
>     Since the request is on a slave node and the processing is done on
> master node, how do you check for the completion of the
> deployment/undeployment of processes and respond back to the client since
> the web service call is a synchronous operation. As DeploymentPoller is
> taking a lot of time in processing, your request will time out right.
>
>
>>
>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl. Each
>> _deploymentUnits stores only what its server has deployed. So think, that
>> a
>> master node goes down another master node appears.But its
>> __deploymentUnits
>> does not have dus which has deployed by the earlier master node. Hence it
>> will not be able retire earlier version of the process which is deployed
>> by
>> previous master. So there will two process which are in "ACTIVE" state
>
>
>> 7) To avoid this, I add the ODEServer as an Observer to check when a new
>> master is electing, then load all the deployment units out of the store.
>> So
>> new master node can have all the dus and can retire appropriate version.
>> Usually loadAll() is called at the server start-up time. But there is no
>> other way to solve this. I tried to use Hazelcast IMap to store all dus
>> among all nodes. But it wasn't success as du is not serializable object.
>>
>
>> 8) I figured out that we do not need send cluster message to others as all
>> the dus' data are persisted to the shared DB. So each node can take the du
>> and retrieve necessary data using already implemented methods in Process
>> Store.
>>
>> 9) But there is an another problem.The axis2 service corresponding to a
>> deployed process does not appear on all nodes of the cluster. That is
>> because each server add du which is deployed by it to the process
>> store.That is why I had use loadAll() when masters are changing. How to
>> solve this?
>>
>
>     I do appriciate your efforts in understanding the implmentation and
> changes that need to be done. You are bang on it.
>     fireEvent(..) is the method that triggers process activation and
> necessary service creation.
>
>
>     But with this given apporach from steps 6 to 8, ODE cannot have
> atleast 2 Active servers for Load balancing. You are concentrating on only
> one active node that will do deployments and cater to process invocations.
>     We should also think about scaling ODE to multiple servers to handle
> load.
>
>     What do you think.
>
>
>> Thank you,
>> Sudharma
>>
>> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
>>
>> > Sudharma,
>> >
>> > Any updates?
>> >
>> > regards,
>> > sathwik
>> >
>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com>
>> wrote:
>> >
>> > > Sudharma,
>> > >
>> > > Can you elaborate on your option 1).
>> > >
>> > > Response to your option 2).
>> > >
>> > >     Process Store is the component that handles process metadata,
>> > > compilation and deployment in ODE. Integration layers in ODE (Axis2,
>> JBI)
>> > > use the process store.
>> > >     Future implementations of IL for ODE will also use the process
>> store.
>> > > We should not be thinking of moving the process store functionality to
>> > the
>> > > integration layers.
>> > >
>> > >
>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
>> > > suba.11@cse.mrt.ac.lk> wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> I understood the problem within dynamic master/slave configuration.
>> In
>> > my
>> > >> approach, when a deployment request is routed to a slave node there
>> will
>> > >> not be a deployment. I suggest two options to avoid it.
>> > >> 1) Have static master/slave configuration only for deploy process
>> > >>
>> > > 2) Modify the deployment web service to complie and verify the process
>> > and
>> > >> then copy it to the deploy folder irrespective of whether its a
>> master
>> > or
>> > >> slave, then deployment poller should take care of the deployment
>> > >>
>> > >>
>> > >
>> > >>
>> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
>> > >>
>> > >> > Sudharma,
>> > >> >
>> > >> > We definitely need a master/slave in the hazelcast cluster. This is
>> > >> > probably needed for the job migration in the Scheduler to migrate
>> the
>> > >> jobs
>> > >> > associated with a down node. Let hold on this topic for future
>> > >> discussion.
>> > >> >
>> > >> > Going by the explanation where the master/slave nodes have certain
>> > >> > predefined tasks to perform is perfectly fine.
>> > >> >
>> > >> > I have this scenario,
>> > >> >
>> > >> > I am using HAProxy as my load balancer and configured 3 nodes in
>> the
>> > >> > cluster.
>> > >> >
>> > >> > Node1 - Active
>> > >> > Node2 - Active
>> > >> > Node3 - Backup
>> > >> >
>> > >> > Load balancing algorithm: RoundRobin
>> > >> >
>> > >> > A Backup node (Node3) is one which the load balancer will not route
>> > >> > requests to, until one of the Active node i.e either Node1 or Node2
>> > has
>> > >> > gone down.
>> > >> >
>> > >> > All these 3 nodes are also part of the hazelcast cluster as well.
>> > >> >
>> > >> > In the hazelcast cluster, assume Node1 is elected as the
>> leader/master
>> > >> and
>> > >> > Node2,Node3 as slaves.
>> > >> >
>> > >> > I initiate the deploy operation on the DeploymentWebService which
>> the
>> > >> load
>> > >> > balancer routes it to one of the Active nodes in the cluster, lets
>> say
>> > >> it's
>> > >> > the Node1. Since Node1 is also the master in the hazelcast cluster,
>> > >> > deployment is a success.
>> > >> >
>> > >> > I initiate another deploy operation on the DeploymentWebService
>> which
>> > >> the
>> > >> > load balancer routes it to the next active node which is Node2.
>> Since
>> > >> Node2
>> > >> > is a slave in the Hazelcast cluster, What happens to the
>> deployment?
>> > >> >
>> > >> > regards,
>> > >> > sathwik
>> > >> >
>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>> > >> > suba.11@cse.mrt.ac.lk
>> > >> > > wrote:
>> > >> >
>> > >> > > Hi,
>> > >> > >
>> > >> > > I will explain my approach as much as possible. The oldest node
>> in
>> > the
>> > >> > > hazelcast cluster is elected as the master node. In the failure
>> of
>> > the
>> > >> > > master node, next oldest node will be elected as the master node.
>> > This
>> > >> > > master-slave configuration is just for deployment. When the
>> > hazelcast
>> > >> > > cluster elected the master node, that node becomes a master node
>> for
>> > >> > > deploying process. So it will do the deploying artifacts. If you
>> > want
>> > >> to
>> > >> > > get the idea of electing master node please refer the code which
>> I
>> > >> have
>> > >> > > located in the github. (
>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>> > >> > >
>> > >> > > I identified separated actions which should be followed by the
>> > master
>> > >> and
>> > >> > > salve nodes.
>> > >> > > Actions which are followed by master node only
>> > >> > > 1) create deployment unit
>> > >> > > 2) set the version nu to deployment unit
>> > >> > > 3) compile deployment unit
>> > >> > > 4) scan deployment unit
>> > >> > > 5) retire previous versions
>> > >> > > Master node and slave nodes should create _processes which stores
>> > >> > > ProcessConfImpl
>> > >> > > Only master node will write the version nu to database, create
>> > >> .deployed
>> > >> > > file
>> > >> > >
>> > >> > > So there are some actions which should be followed only by master
>> > node
>> > >> > > while other actions should be followed by all the nodes.The idea
>> of
>> > >> > having
>> > >> > > a master node is deploying artifacts and avoid others from
>> writing
>> > the
>> > >> > > version nu to database.
>> > >> > > Whether a node is active or passive, all nodes should do the
>> > >> > > deployment.Master
>> > >> > > and slaves will follow necessary actions as in above.
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com>
>> wrote:
>> > >> > >
>> > >> > > > Nandika,
>> > >> > > >
>> > >> > > > I very well understand what you have put across, but it's
>> > secondary
>> > >> to
>> > >> > me
>> > >> > > > now.
>> > >> > > >
>> > >> > > > Sudharma,
>> > >> > > > My primary concern is to understand at a high level the
>> deployment
>> > >> > > > architecture and how would master-slave configuration fit in.
>> Are
>> > >> there
>> > >> > > any
>> > >> > > > restrictions imposed by the in-progress design?
>> > >> > > >
>> > >> > > > Firstly, how would ODE process deployment work under these
>> cluster
>> > >> > > > configurations?
>> > >> > > >
>> > >> > > > Sample Cluster configurations: A load balancer is frontending
>> the
>> > >> > > servers.
>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
>> Active or
>> > >> > > Passive.
>> > >> > > >
>> > >> > > > regards,
>> > >> > > > sathwik
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>> > >> > jayawark@gmail.com
>> > >> > > >
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Hi Sathwik,
>> > >> > > > >
>> > >> > > > > According to my understanding, in the clustering scenario,
>> the
>> > >> master
>> > >> > > > node
>> > >> > > > > should perform all the deployment actions and the slave nodes
>> > also
>> > >> > need
>> > >> > > > to
>> > >> > > > > perform some deployment actions. For example, the slave nodes
>> > also
>> > >> > > should
>> > >> > > > > handle the process ACTIVATED event so that the process
>> > >> configuration
>> > >> > is
>> > >> > > > > added to the engine and necessary web services are created so
>> > that
>> > >> > when
>> > >> > > > the
>> > >> > > > > load balancer send requests to any node in the cluster, it is
>> > >> ready
>> > >> > to
>> > >> > > > > accept those requests.
>> > >> > > > >
>> > >> > > > > Regards
>> > >> > > > > Nandika
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>> > >> sathwik.bp@gmail.com>
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > Sudharma,
>> > >> > > > > >
>> > >> > > > > > Where are you going to configure the master-slaves, is it
>> in
>> > the
>> > >> > web
>> > >> > > > > > application level or at the load balancer?
>> > >> > > > > >
>> > >> > > > > > regards,
>> > >> > > > > > sathwik
>> > >> > > > > >
>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
>> > >> > > > > > suba.11@cse.mrt.ac.lk>
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Hi Tammo,
>> > >> > > > > > >
>> > >> > > > > > > Can you suggest the best method from these to implement?
>> As
>> > >> > first I
>> > >> > > > > > > suggested the master-slaves scenario I think it is easy
>> to
>> > >> > > implement
>> > >> > > > > than
>> > >> > > > > > > distributed lock scenario. However if you can suggest one
>> > from
>> > >> > > these
>> > >> > > > > two,
>> > >> > > > > > > then I can think about it.
>> > >> > > > > > >
>> > >> > > > > > > Thank you
>> > >> > > > > > >
>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
>> sathwik.bp@gmail.com>
>> > >> > wrote:
>> > >> > > > > > >
>> > >> > > > > > > > With respect to the hotdeployment,
>> > >> > > > > > > >
>> > >> > > > > > > > We can drop the deployment archive onto the deployment
>> > >> folder.
>> > >> > > > Since
>> > >> > > > > > the
>> > >> > > > > > > > DeploymentPoller are acquiring the distributed lock for
>> > the
>> > >> > > > > > > DeploymentUnit,
>> > >> > > > > > > > only one of the nodes will get the lock and initiate
>> the
>> > >> > > > deployment.
>> > >> > > > > > > > DeploymentPollers on other nodes will fail in acquiring
>> > the
>> > >> > lock
>> > >> > > > and
>> > >> > > > > > > hence
>> > >> > > > > > > > will silently ignore it.
>> > >> > > > > > > >
>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>> > >> > > > sathwik.bp@gmail.com>
>> > >> > > > > > > > wrote:
>> > >> > > > > > > >
>> > >> > > > > > > > > Hi Tammo,
>> > >> > > > > > > > >
>> > >> > > > > > > > > The distributed lock acquisition on the
>> DeploymentUnit
>> > >> should
>> > >> > > be
>> > >> > > > > > added
>> > >> > > > > > > to
>> > >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
>> > >> > > > > > > > >
>> > >> > > > > > > > > When a deployment operation is initiated through the
>> > >> > > > > > > > DeploymentWebService,
>> > >> > > > > > > > > The load balancer routes it to any of the available
>> > nodes.
>> > >> > > > > > > > >
>> > >> > > > > > > > > On the routed node, the DeploymentWebService acquires
>> > the
>> > >> > > > > Distributed
>> > >> > > > > > > > > lock. On the remaining nodes the DeploymentPoller
>> will
>> > >> try to
>> > >> > > > > acquire
>> > >> > > > > > > the
>> > >> > > > > > > > > distributed lock and will not get it and hence will
>> > >> silently
>> > >> > > > ignore
>> > >> > > > > > it.
>> > >> > > > > > > > >
>> > >> > > > > > > > > Once the routed node completes the deployment, it
>> will
>> > >> > release
>> > >> > > > the
>> > >> > > > > > > lock.
>> > >> > > > > > > > > This way we don't have to stall the DeploymentPoller
>> in
>> > >> other
>> > >> > > > > nodes.
>> > >> > > > > > > > >
>> > >> > > > > > > > > Does it answer the concerns?
>> > >> > > > > > > > >
>> > >> > > > > > > > >
>> > >> > > > > > > > > Now, if we give the responsibility of identifying the
>> > >> master
>> > >> > > node
>> > >> > > > > to
>> > >> > > > > > > the
>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
>> balancer
>> > to
>> > >> > > change
>> > >> > > > > > it's
>> > >> > > > > > > > > configuration about the master node?
>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
>> > >> > > > > > > > > node1 -master
>> > >> > > > > > > > > node2 - slave
>> > >> > > > > > > > > node3 - slave
>> > >> > > > > > > > >
>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as master
>> > node,
>> > >> > but
>> > >> > > > > > > hazelcast
>> > >> > > > > > > > > might promote Node3 as master node. They are out of
>> > sync.
>> > >> > > > > > > > >
>> > >> > > > > > > > > Is this argument valid?
>> > >> > > > > > > > >
>> > >> > > > > > > > > regards,
>> > >> > > > > > > > > sathwik
>> > >> > > > > > > > >
>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
>> > >> > > > > > > tvanlessen@gmail.com>
>> > >> > > > > > > > > wrote:
>> > >> > > > > > > > >
>> > >> > > > > > > > >> Hi Sudharma,
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> what do you expect from the "other nodes
>> deployment"?
>> > >> > > > Compilation
>> > >> > > > > is
>> > >> > > > > > > not
>> > >> > > > > > > > >> needed since the CBP file is written to the (shared)
>> > FS.
>> > >> > > > > > Registration
>> > >> > > > > > > is
>> > >> > > > > > > > >> also not needed, since it is done via the shared
>> > >> database.
>> > >> > So
>> > >> > > > the
>> > >> > > > > > only
>> > >> > > > > > > > >> thing that might be needed is to tell the engine
>> that
>> > >> there
>> > >> > > is a
>> > >> > > > > new
>> > >> > > > > > > > >> deployment. I'd need to check that. If this is
>> needed,
>> > I
>> > >> > > revert
>> > >> > > > my
>> > >> > > > > > > last
>> > >> > > > > > > > >> statement, then it is perhaps better to just send an
>> > >> event
>> > >> > > over
>> > >> > > > > > > > Hazelcast
>> > >> > > > > > > > >> to all nodes that the deployment has changed.
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> Best,
>> > >> > > > > > > > >>   Tammo
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
>> subasinghe <
>> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
>> > >> > > > > > > > >> > wrote:
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> > Hi Tammo,
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > The master node writes meta data. But runtime
>> > >> information
>> > >> > > must
>> > >> > > > > be
>> > >> > > > > > > > >> available
>> > >> > > > > > > > >> > in all nodes.Since the folder is shared, all nodes
>> > will
>> > >> > see
>> > >> > > > the
>> > >> > > > > > > > >> > availability of a new process. My idea is for
>> master
>> > >> node
>> > >> > to
>> > >> > > > > write
>> > >> > > > > > > the
>> > >> > > > > > > > >> meta
>> > >> > > > > > > > >> > data and other nodes to just read the meta data
>> and
>> > >> load
>> > >> > > > > > process.So
>> > >> > > > > > > we
>> > >> > > > > > > > >> need
>> > >> > > > > > > > >> > a small delay between master node deployment and
>> > other
>> > >> > nodes
>> > >> > > > > > > > deployment.
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Is there anyway to set the delay between master
>> node
>> > >> and
>> > >> > > > slaves
>> > >> > > > > > > until
>> > >> > > > > > > > >> > master node finish the deployment?
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Thank you
>> > >> > > > > > > > >> > Sudharma
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
>> > >> > > > tvanlessen@gmail.com
>> > >> > > > > >
>> > >> > > > > > > > wrote:
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > > Hi Sathwik,
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
>> > >> > > > > > > sathwik.bp@gmail.com>
>> > >> > > > > > > > >> > wrote:
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > > Sudharma/Tammo,
>> > >> > > > > > > > >> > > >
>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is the
>> master
>> > >> node
>> > >> > in
>> > >> > > > the
>> > >> > > > > > > > cluster?
>> > >> > > > > > > > >> > > >
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > I think the easiest approach is to always elect
>> the
>> > >> > oldest
>> > >> > > > > node
>> > >> > > > > > in
>> > >> > > > > > > > the
>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can
>> > easily
>> > >> > asked
>> > >> > > > for
>> > >> > > > > > > this
>> > >> > > > > > > > >> > > information.
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment
>> Pollers
>> > in
>> > >> > the
>> > >> > > > > slave
>> > >> > > > > > > > nodes?
>> > >> > > > > > > > >> > > >
>> > >> > > > > > > > >> > > >
>> > >> > > > > > > > >> > > Absolutely.
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > Suggestion:
>> > >> > > > > > > > >> > > > I am not sure whether do we need
>> Master-SLaves.
>> > Why
>> > >> > not
>> > >> > > > give
>> > >> > > > > > > every
>> > >> > > > > > > > >> node
>> > >> > > > > > > > >> > > in
>> > >> > > > > > > > >> > > > the cluster the same status (Active-Active).
>> > >> > > > > > > > >> > > >
>> > >> > > > > > > > >> > > > When a new deployment is made, the load
>> balancer
>> > >> can
>> > >> > > push
>> > >> > > > it
>> > >> > > > > > to
>> > >> > > > > > > > any
>> > >> > > > > > > > >> of
>> > >> > > > > > > > >> > > the
>> > >> > > > > > > > >> > > > available nodes. That node will probably
>> acquire
>> > a
>> > >> > > > > distributed
>> > >> > > > > > > > lock
>> > >> > > > > > > > >> on
>> > >> > > > > > > > >> > > the
>> > >> > > > > > > > >> > > > deployment unit and acts as master for that
>> > >> > deployment.
>> > >> > > > This
>> > >> > > > > > > > ensures
>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes. Probably
>> no
>> > >> static
>> > >> > > > > > > > >> configuration of
>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
>> > >> > hazelcast.
>> > >> > > > > > > > >> > > >
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > But this would not allow to have the
>> hotdeployment
>> > >> via
>> > >> > > > > > filesystem
>> > >> > > > > > > > >> still
>> > >> > > > > > > > >> > > enabled, right?
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > Best,
>> > >> > > > > > > > >> > >   Tammo
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> > > --
>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>> > >> > > > > > > > >> > >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > > >>
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> --
>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>> > >> > > > > > > > >>
>> > >> > > > > > > > >
>> > >> > > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

find response inline

On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <su...@cse.mrt.ac.lk>
wrote:

> Hi,
>
> Sorry for the late reply. I was trying to achieve a proper solution.
> Following is my approach.
>
> 1) I elected master node for deploying purpose. So when a master node goes
> down hazelcast will elect the next oldest node for master.
>

    Perfect


> 2) ODEServer can identify whether clustering is enabled or not by getting
> the property value in ode-axis2.properties file. So I introduced new
> property called "ode-axis2.hazelcast.clustering.enabled". If there is no
> clustering enabled server will work as it is. If clustering is enabled,
> cluster will be initialized.
>

    Perfect


> 3) In manual deployment its responsibility is taken by the
> DeploymentPoller. So I give the deployment capability to master node,s
> poller by setting "isDeploymentFromODEFileSystemAllowed()" to true.So
> others will not be able to go into check() method in DeploymentPoller. So
> deployment will be done by only the master node.
>

    Perfect


>
> 4) In DeploymentWebService, I had to consider few cases.If the deploy
> request goes to the master node, it will deploy the process through web
> service.Others pollers will not go into check() method as they are not
> masters. So master can continue without any involvement of others.
>
>
    Perfect


> 5) If the deploy request goes to a slave node, it will do up to file
> creation in the file system.Slave will be stopped at that point. As only
> master poller is checking, master can continue from created files in the
> file system.
>

    DeploymentWebService provides synchronous operations. The status of the
operation should be communicated to the calling client in the same call.
    DeploymentPoller is a backend thread that goes over each and every
directory under the Deployment directory checking for any changes to
deploy.xml in existing processes and deploy newly added processes. This
process is  sequential and time consuming. As the process directories
grows, so does the time taken for execution of the thread.

    Since the request is on a slave node and the processing is done on
master node, how do you check for the completion of the
deployment/undeployment of processes and respond back to the client since
the web service call is a synchronous operation. As DeploymentPoller is
taking a lot of time in processing, your request will time out right.


>
> 6) But there was problem with _deploymentUnits in ProcessStoreImpl. Each
> _deploymentUnits stores only what its server has deployed. So think, that a
> master node goes down another master node appears.But its __deploymentUnits
> does not have dus which has deployed by the earlier master node. Hence it
> will not be able retire earlier version of the process which is deployed by
> previous master. So there will two process which are in "ACTIVE" state


> 7) To avoid this, I add the ODEServer as an Observer to check when a new
> master is electing, then load all the deployment units out of the store. So
> new master node can have all the dus and can retire appropriate version.
> Usually loadAll() is called at the server start-up time. But there is no
> other way to solve this. I tried to use Hazelcast IMap to store all dus
> among all nodes. But it wasn't success as du is not serializable object.
>

> 8) I figured out that we do not need send cluster message to others as all
> the dus' data are persisted to the shared DB. So each node can take the du
> and retrieve necessary data using already implemented methods in Process
> Store.
>
> 9) But there is an another problem.The axis2 service corresponding to a
> deployed process does not appear on all nodes of the cluster. That is
> because each server add du which is deployed by it to the process
> store.That is why I had use loadAll() when masters are changing. How to
> solve this?
>

    I do appriciate your efforts in understanding the implmentation and
changes that need to be done. You are bang on it.
    fireEvent(..) is the method that triggers process activation and
necessary service creation.


    But with this given apporach from steps 6 to 8, ODE cannot have atleast
2 Active servers for Load balancing. You are concentrating on only one
active node that will do deployments and cater to process invocations.
    We should also think about scaling ODE to multiple servers to handle
load.

    What do you think.


> Thank you,
> Sudharma
>
> On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:
>
> > Sudharma,
> >
> > Any updates?
> >
> > regards,
> > sathwik
> >
> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com>
> wrote:
> >
> > > Sudharma,
> > >
> > > Can you elaborate on your option 1).
> > >
> > > Response to your option 2).
> > >
> > >     Process Store is the component that handles process metadata,
> > > compilation and deployment in ODE. Integration layers in ODE (Axis2,
> JBI)
> > > use the process store.
> > >     Future implementations of IL for ODE will also use the process
> store.
> > > We should not be thinking of moving the process store functionality to
> > the
> > > integration layers.
> > >
> > >
> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> > > suba.11@cse.mrt.ac.lk> wrote:
> > >
> > >> Hi,
> > >>
> > >> I understood the problem within dynamic master/slave configuration. In
> > my
> > >> approach, when a deployment request is routed to a slave node there
> will
> > >> not be a deployment. I suggest two options to avoid it.
> > >> 1) Have static master/slave configuration only for deploy process
> > >>
> > > 2) Modify the deployment web service to complie and verify the process
> > and
> > >> then copy it to the deploy folder irrespective of whether its a master
> > or
> > >> slave, then deployment poller should take care of the deployment
> > >>
> > >>
> > >
> > >>
> > >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
> > >>
> > >> > Sudharma,
> > >> >
> > >> > We definitely need a master/slave in the hazelcast cluster. This is
> > >> > probably needed for the job migration in the Scheduler to migrate
> the
> > >> jobs
> > >> > associated with a down node. Let hold on this topic for future
> > >> discussion.
> > >> >
> > >> > Going by the explanation where the master/slave nodes have certain
> > >> > predefined tasks to perform is perfectly fine.
> > >> >
> > >> > I have this scenario,
> > >> >
> > >> > I am using HAProxy as my load balancer and configured 3 nodes in the
> > >> > cluster.
> > >> >
> > >> > Node1 - Active
> > >> > Node2 - Active
> > >> > Node3 - Backup
> > >> >
> > >> > Load balancing algorithm: RoundRobin
> > >> >
> > >> > A Backup node (Node3) is one which the load balancer will not route
> > >> > requests to, until one of the Active node i.e either Node1 or Node2
> > has
> > >> > gone down.
> > >> >
> > >> > All these 3 nodes are also part of the hazelcast cluster as well.
> > >> >
> > >> > In the hazelcast cluster, assume Node1 is elected as the
> leader/master
> > >> and
> > >> > Node2,Node3 as slaves.
> > >> >
> > >> > I initiate the deploy operation on the DeploymentWebService which
> the
> > >> load
> > >> > balancer routes it to one of the Active nodes in the cluster, lets
> say
> > >> it's
> > >> > the Node1. Since Node1 is also the master in the hazelcast cluster,
> > >> > deployment is a success.
> > >> >
> > >> > I initiate another deploy operation on the DeploymentWebService
> which
> > >> the
> > >> > load balancer routes it to the next active node which is Node2.
> Since
> > >> Node2
> > >> > is a slave in the Hazelcast cluster, What happens to the deployment?
> > >> >
> > >> > regards,
> > >> > sathwik
> > >> >
> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> > >> > suba.11@cse.mrt.ac.lk
> > >> > > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I will explain my approach as much as possible. The oldest node in
> > the
> > >> > > hazelcast cluster is elected as the master node. In the failure of
> > the
> > >> > > master node, next oldest node will be elected as the master node.
> > This
> > >> > > master-slave configuration is just for deployment. When the
> > hazelcast
> > >> > > cluster elected the master node, that node becomes a master node
> for
> > >> > > deploying process. So it will do the deploying artifacts. If you
> > want
> > >> to
> > >> > > get the idea of electing master node please refer the code which I
> > >> have
> > >> > > located in the github. (
> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
> > >> > >
> > >> > > I identified separated actions which should be followed by the
> > master
> > >> and
> > >> > > salve nodes.
> > >> > > Actions which are followed by master node only
> > >> > > 1) create deployment unit
> > >> > > 2) set the version nu to deployment unit
> > >> > > 3) compile deployment unit
> > >> > > 4) scan deployment unit
> > >> > > 5) retire previous versions
> > >> > > Master node and slave nodes should create _processes which stores
> > >> > > ProcessConfImpl
> > >> > > Only master node will write the version nu to database, create
> > >> .deployed
> > >> > > file
> > >> > >
> > >> > > So there are some actions which should be followed only by master
> > node
> > >> > > while other actions should be followed by all the nodes.The idea
> of
> > >> > having
> > >> > > a master node is deploying artifacts and avoid others from writing
> > the
> > >> > > version nu to database.
> > >> > > Whether a node is active or passive, all nodes should do the
> > >> > > deployment.Master
> > >> > > and slaves will follow necessary actions as in above.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com>
> wrote:
> > >> > >
> > >> > > > Nandika,
> > >> > > >
> > >> > > > I very well understand what you have put across, but it's
> > secondary
> > >> to
> > >> > me
> > >> > > > now.
> > >> > > >
> > >> > > > Sudharma,
> > >> > > > My primary concern is to understand at a high level the
> deployment
> > >> > > > architecture and how would master-slave configuration fit in.
> Are
> > >> there
> > >> > > any
> > >> > > > restrictions imposed by the in-progress design?
> > >> > > >
> > >> > > > Firstly, how would ODE process deployment work under these
> cluster
> > >> > > > configurations?
> > >> > > >
> > >> > > > Sample Cluster configurations: A load balancer is frontending
> the
> > >> > > servers.
> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in Active
> or
> > >> > > Passive.
> > >> > > >
> > >> > > > regards,
> > >> > > > sathwik
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> > >> > jayawark@gmail.com
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Sathwik,
> > >> > > > >
> > >> > > > > According to my understanding, in the clustering scenario, the
> > >> master
> > >> > > > node
> > >> > > > > should perform all the deployment actions and the slave nodes
> > also
> > >> > need
> > >> > > > to
> > >> > > > > perform some deployment actions. For example, the slave nodes
> > also
> > >> > > should
> > >> > > > > handle the process ACTIVATED event so that the process
> > >> configuration
> > >> > is
> > >> > > > > added to the engine and necessary web services are created so
> > that
> > >> > when
> > >> > > > the
> > >> > > > > load balancer send requests to any node in the cluster, it is
> > >> ready
> > >> > to
> > >> > > > > accept those requests.
> > >> > > > >
> > >> > > > > Regards
> > >> > > > > Nandika
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> > >> sathwik.bp@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Sudharma,
> > >> > > > > >
> > >> > > > > > Where are you going to configure the master-slaves, is it in
> > the
> > >> > web
> > >> > > > > > application level or at the load balancer?
> > >> > > > > >
> > >> > > > > > regards,
> > >> > > > > > sathwik
> > >> > > > > >
> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > >> > > > > > suba.11@cse.mrt.ac.lk>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Tammo,
> > >> > > > > > >
> > >> > > > > > > Can you suggest the best method from these to implement?
> As
> > >> > first I
> > >> > > > > > > suggested the master-slaves scenario I think it is easy to
> > >> > > implement
> > >> > > > > than
> > >> > > > > > > distributed lock scenario. However if you can suggest one
> > from
> > >> > > these
> > >> > > > > two,
> > >> > > > > > > then I can think about it.
> > >> > > > > > >
> > >> > > > > > > Thank you
> > >> > > > > > >
> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
> sathwik.bp@gmail.com>
> > >> > wrote:
> > >> > > > > > >
> > >> > > > > > > > With respect to the hotdeployment,
> > >> > > > > > > >
> > >> > > > > > > > We can drop the deployment archive onto the deployment
> > >> folder.
> > >> > > > Since
> > >> > > > > > the
> > >> > > > > > > > DeploymentPoller are acquiring the distributed lock for
> > the
> > >> > > > > > > DeploymentUnit,
> > >> > > > > > > > only one of the nodes will get the lock and initiate the
> > >> > > > deployment.
> > >> > > > > > > > DeploymentPollers on other nodes will fail in acquiring
> > the
> > >> > lock
> > >> > > > and
> > >> > > > > > > hence
> > >> > > > > > > > will silently ignore it.
> > >> > > > > > > >
> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> > >> > > > sathwik.bp@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Tammo,
> > >> > > > > > > > >
> > >> > > > > > > > > The distributed lock acquisition on the DeploymentUnit
> > >> should
> > >> > > be
> > >> > > > > > added
> > >> > > > > > > to
> > >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
> > >> > > > > > > > >
> > >> > > > > > > > > When a deployment operation is initiated through the
> > >> > > > > > > > DeploymentWebService,
> > >> > > > > > > > > The load balancer routes it to any of the available
> > nodes.
> > >> > > > > > > > >
> > >> > > > > > > > > On the routed node, the DeploymentWebService acquires
> > the
> > >> > > > > Distributed
> > >> > > > > > > > > lock. On the remaining nodes the DeploymentPoller will
> > >> try to
> > >> > > > > acquire
> > >> > > > > > > the
> > >> > > > > > > > > distributed lock and will not get it and hence will
> > >> silently
> > >> > > > ignore
> > >> > > > > > it.
> > >> > > > > > > > >
> > >> > > > > > > > > Once the routed node completes the deployment, it will
> > >> > release
> > >> > > > the
> > >> > > > > > > lock.
> > >> > > > > > > > > This way we don't have to stall the DeploymentPoller
> in
> > >> other
> > >> > > > > nodes.
> > >> > > > > > > > >
> > >> > > > > > > > > Does it answer the concerns?
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Now, if we give the responsibility of identifying the
> > >> master
> > >> > > node
> > >> > > > > to
> > >> > > > > > > the
> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
> balancer
> > to
> > >> > > change
> > >> > > > > > it's
> > >> > > > > > > > > configuration about the master node?
> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
> > >> > > > > > > > > node1 -master
> > >> > > > > > > > > node2 - slave
> > >> > > > > > > > > node3 - slave
> > >> > > > > > > > >
> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as master
> > node,
> > >> > but
> > >> > > > > > > hazelcast
> > >> > > > > > > > > might promote Node3 as master node. They are out of
> > sync.
> > >> > > > > > > > >
> > >> > > > > > > > > Is this argument valid?
> > >> > > > > > > > >
> > >> > > > > > > > > regards,
> > >> > > > > > > > > sathwik
> > >> > > > > > > > >
> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > >> > > > > > > tvanlessen@gmail.com>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Hi Sudharma,
> > >> > > > > > > > >>
> > >> > > > > > > > >> what do you expect from the "other nodes deployment"?
> > >> > > > Compilation
> > >> > > > > is
> > >> > > > > > > not
> > >> > > > > > > > >> needed since the CBP file is written to the (shared)
> > FS.
> > >> > > > > > Registration
> > >> > > > > > > is
> > >> > > > > > > > >> also not needed, since it is done via the shared
> > >> database.
> > >> > So
> > >> > > > the
> > >> > > > > > only
> > >> > > > > > > > >> thing that might be needed is to tell the engine that
> > >> there
> > >> > > is a
> > >> > > > > new
> > >> > > > > > > > >> deployment. I'd need to check that. If this is
> needed,
> > I
> > >> > > revert
> > >> > > > my
> > >> > > > > > > last
> > >> > > > > > > > >> statement, then it is perhaps better to just send an
> > >> event
> > >> > > over
> > >> > > > > > > > Hazelcast
> > >> > > > > > > > >> to all nodes that the deployment has changed.
> > >> > > > > > > > >>
> > >> > > > > > > > >> Best,
> > >> > > > > > > > >>   Tammo
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
> subasinghe <
> > >> > > > > > > > >> suba.11@cse.mrt.ac.lk
> > >> > > > > > > > >> > wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >> > Hi Tammo,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > The master node writes meta data. But runtime
> > >> information
> > >> > > must
> > >> > > > > be
> > >> > > > > > > > >> available
> > >> > > > > > > > >> > in all nodes.Since the folder is shared, all nodes
> > will
> > >> > see
> > >> > > > the
> > >> > > > > > > > >> > availability of a new process. My idea is for
> master
> > >> node
> > >> > to
> > >> > > > > write
> > >> > > > > > > the
> > >> > > > > > > > >> meta
> > >> > > > > > > > >> > data and other nodes to just read the meta data and
> > >> load
> > >> > > > > > process.So
> > >> > > > > > > we
> > >> > > > > > > > >> need
> > >> > > > > > > > >> > a small delay between master node deployment and
> > other
> > >> > nodes
> > >> > > > > > > > deployment.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Is there anyway to set the delay between master
> node
> > >> and
> > >> > > > slaves
> > >> > > > > > > until
> > >> > > > > > > > >> > master node finish the deployment?
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thank you
> > >> > > > > > > > >> > Sudharma
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> > >> > > > tvanlessen@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > > Hi Sathwik,
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > >> > > > > > > sathwik.bp@gmail.com>
> > >> > > > > > > > >> > wrote:
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > > Sudharma/Tammo,
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > 1) How do we plan to decide which is the master
> > >> node
> > >> > in
> > >> > > > the
> > >> > > > > > > > cluster?
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I think the easiest approach is to always elect
> the
> > >> > oldest
> > >> > > > > node
> > >> > > > > > in
> > >> > > > > > > > the
> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can
> > easily
> > >> > asked
> > >> > > > for
> > >> > > > > > > this
> > >> > > > > > > > >> > > information.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment
> Pollers
> > in
> > >> > the
> > >> > > > > slave
> > >> > > > > > > > nodes?
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > Absolutely.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > Suggestion:
> > >> > > > > > > > >> > > > I am not sure whether do we need Master-SLaves.
> > Why
> > >> > not
> > >> > > > give
> > >> > > > > > > every
> > >> > > > > > > > >> node
> > >> > > > > > > > >> > > in
> > >> > > > > > > > >> > > > the cluster the same status (Active-Active).
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > When a new deployment is made, the load
> balancer
> > >> can
> > >> > > push
> > >> > > > it
> > >> > > > > > to
> > >> > > > > > > > any
> > >> > > > > > > > >> of
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > available nodes. That node will probably
> acquire
> > a
> > >> > > > > distributed
> > >> > > > > > > > lock
> > >> > > > > > > > >> on
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > deployment unit and acts as master for that
> > >> > deployment.
> > >> > > > This
> > >> > > > > > > > ensures
> > >> > > > > > > > >> > > > optimum usage of the cluster nodes. Probably no
> > >> static
> > >> > > > > > > > >> configuration of
> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
> > >> > hazelcast.
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > But this would not allow to have the
> hotdeployment
> > >> via
> > >> > > > > > filesystem
> > >> > > > > > > > >> still
> > >> > > > > > > > >> > > enabled, right?
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > Best,
> > >> > > > > > > > >> > >   Tammo
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > --
> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >> --
> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi,

Sorry for the late reply. I was trying to achieve a proper solution.
Following is my approach.

1) I elected master node for deploying purpose. So when a master node goes
down hazelcast will elect the next oldest node for master.

2) ODEServer can identify whether clustering is enabled or not by getting
the property value in ode-axis2.properties file. So I introduced new
property called "ode-axis2.hazelcast.clustering.enabled". If there is no
clustering enabled server will work as it is. If clustering is enabled,
cluster will be initialized.

3) In manual deployment its responsibility is taken by the
DeploymentPoller. So I give the deployment capability to master node,s
poller by setting "isDeploymentFromODEFileSystemAllowed()" to true.So
others will not be able to go into check() method in DeploymentPoller. So
deployment will be done by only the master node.

4) In DeploymentWebService, I had to consider few cases.If the deploy
request goes to the master node, it will deploy the process through web
service.Others pollers will not go into check() method as they are not
masters. So master can continue without any involvement of others.

5) If the deploy request goes to a slave node, it will do up to file
creation in the file system.Slave will be stopped at that point. As only
master poller is checking, master can continue from created files in the
file system.

6) But there was problem with _deploymentUnits in ProcessStoreImpl. Each
_deploymentUnits stores only what its server has deployed. So think, that a
master node goes down another master node appears.But its __deploymentUnits
does not have dus which has deployed by the earlier master node. Hence it
will not be able retire earlier version of the process which is deployed by
previous master. So there will two process which are in "ACTIVE" state

7) To avoid this, I add the ODEServer as an Observer to check when a new
master is electing, then load all the deployment units out of the store. So
new master node can have all the dus and can retire appropriate version.
Usually loadAll() is called at the server start-up time. But there is no
other way to solve this. I tried to use Hazelcast IMap to store all dus
among all nodes. But it wasn't success as du is not serializable object.

8) I figured out that we do not need send cluster message to others as all
the dus' data are persisted to the shared DB. So each node can take the du
and retrieve necessary data using already implemented methods in Process
Store.

9) But there is an another problem.The axis2 service corresponding to a
deployed process does not appear on all nodes of the cluster. That is
because each server add du which is deployed by it to the process
store.That is why I had use loadAll() when masters are changing. How to
solve this?

Thank you,
Sudharma

On 2 June 2015 at 08:51, Sathwik B P <sa...@gmail.com> wrote:

> Sudharma,
>
> Any updates?
>
> regards,
> sathwik
>
> On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com> wrote:
>
> > Sudharma,
> >
> > Can you elaborate on your option 1).
> >
> > Response to your option 2).
> >
> >     Process Store is the component that handles process metadata,
> > compilation and deployment in ODE. Integration layers in ODE (Axis2, JBI)
> > use the process store.
> >     Future implementations of IL for ODE will also use the process store.
> > We should not be thinking of moving the process store functionality to
> the
> > integration layers.
> >
> >
> > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> > suba.11@cse.mrt.ac.lk> wrote:
> >
> >> Hi,
> >>
> >> I understood the problem within dynamic master/slave configuration. In
> my
> >> approach, when a deployment request is routed to a slave node there will
> >> not be a deployment. I suggest two options to avoid it.
> >> 1) Have static master/slave configuration only for deploy process
> >>
> > 2) Modify the deployment web service to complie and verify the process
> and
> >> then copy it to the deploy folder irrespective of whether its a master
> or
> >> slave, then deployment poller should take care of the deployment
> >>
> >>
> >
> >>
> >> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
> >>
> >> > Sudharma,
> >> >
> >> > We definitely need a master/slave in the hazelcast cluster. This is
> >> > probably needed for the job migration in the Scheduler to migrate the
> >> jobs
> >> > associated with a down node. Let hold on this topic for future
> >> discussion.
> >> >
> >> > Going by the explanation where the master/slave nodes have certain
> >> > predefined tasks to perform is perfectly fine.
> >> >
> >> > I have this scenario,
> >> >
> >> > I am using HAProxy as my load balancer and configured 3 nodes in the
> >> > cluster.
> >> >
> >> > Node1 - Active
> >> > Node2 - Active
> >> > Node3 - Backup
> >> >
> >> > Load balancing algorithm: RoundRobin
> >> >
> >> > A Backup node (Node3) is one which the load balancer will not route
> >> > requests to, until one of the Active node i.e either Node1 or Node2
> has
> >> > gone down.
> >> >
> >> > All these 3 nodes are also part of the hazelcast cluster as well.
> >> >
> >> > In the hazelcast cluster, assume Node1 is elected as the leader/master
> >> and
> >> > Node2,Node3 as slaves.
> >> >
> >> > I initiate the deploy operation on the DeploymentWebService which the
> >> load
> >> > balancer routes it to one of the Active nodes in the cluster, lets say
> >> it's
> >> > the Node1. Since Node1 is also the master in the hazelcast cluster,
> >> > deployment is a success.
> >> >
> >> > I initiate another deploy operation on the DeploymentWebService which
> >> the
> >> > load balancer routes it to the next active node which is Node2. Since
> >> Node2
> >> > is a slave in the Hazelcast cluster, What happens to the deployment?
> >> >
> >> > regards,
> >> > sathwik
> >> >
> >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> >> > suba.11@cse.mrt.ac.lk
> >> > > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I will explain my approach as much as possible. The oldest node in
> the
> >> > > hazelcast cluster is elected as the master node. In the failure of
> the
> >> > > master node, next oldest node will be elected as the master node.
> This
> >> > > master-slave configuration is just for deployment. When the
> hazelcast
> >> > > cluster elected the master node, that node becomes a master node for
> >> > > deploying process. So it will do the deploying artifacts. If you
> want
> >> to
> >> > > get the idea of electing master node please refer the code which I
> >> have
> >> > > located in the github. (
> >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
> >> > >
> >> > > I identified separated actions which should be followed by the
> master
> >> and
> >> > > salve nodes.
> >> > > Actions which are followed by master node only
> >> > > 1) create deployment unit
> >> > > 2) set the version nu to deployment unit
> >> > > 3) compile deployment unit
> >> > > 4) scan deployment unit
> >> > > 5) retire previous versions
> >> > > Master node and slave nodes should create _processes which stores
> >> > > ProcessConfImpl
> >> > > Only master node will write the version nu to database, create
> >> .deployed
> >> > > file
> >> > >
> >> > > So there are some actions which should be followed only by master
> node
> >> > > while other actions should be followed by all the nodes.The idea of
> >> > having
> >> > > a master node is deploying artifacts and avoid others from writing
> the
> >> > > version nu to database.
> >> > > Whether a node is active or passive, all nodes should do the
> >> > > deployment.Master
> >> > > and slaves will follow necessary actions as in above.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com> wrote:
> >> > >
> >> > > > Nandika,
> >> > > >
> >> > > > I very well understand what you have put across, but it's
> secondary
> >> to
> >> > me
> >> > > > now.
> >> > > >
> >> > > > Sudharma,
> >> > > > My primary concern is to understand at a high level the deployment
> >> > > > architecture and how would master-slave configuration fit in. Are
> >> there
> >> > > any
> >> > > > restrictions imposed by the in-progress design?
> >> > > >
> >> > > > Firstly, how would ODE process deployment work under these cluster
> >> > > > configurations?
> >> > > >
> >> > > > Sample Cluster configurations: A load balancer is frontending the
> >> > > servers.
> >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> >> > > > 3) Cluster with 2+ nodes with additional nodes either in Active or
> >> > > Passive.
> >> > > >
> >> > > > regards,
> >> > > > sathwik
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> >> > jayawark@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Sathwik,
> >> > > > >
> >> > > > > According to my understanding, in the clustering scenario, the
> >> master
> >> > > > node
> >> > > > > should perform all the deployment actions and the slave nodes
> also
> >> > need
> >> > > > to
> >> > > > > perform some deployment actions. For example, the slave nodes
> also
> >> > > should
> >> > > > > handle the process ACTIVATED event so that the process
> >> configuration
> >> > is
> >> > > > > added to the engine and necessary web services are created so
> that
> >> > when
> >> > > > the
> >> > > > > load balancer send requests to any node in the cluster, it is
> >> ready
> >> > to
> >> > > > > accept those requests.
> >> > > > >
> >> > > > > Regards
> >> > > > > Nandika
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> >> sathwik.bp@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Sudharma,
> >> > > > > >
> >> > > > > > Where are you going to configure the master-slaves, is it in
> the
> >> > web
> >> > > > > > application level or at the load balancer?
> >> > > > > >
> >> > > > > > regards,
> >> > > > > > sathwik
> >> > > > > >
> >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> >> > > > > > suba.11@cse.mrt.ac.lk>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Tammo,
> >> > > > > > >
> >> > > > > > > Can you suggest the best method from these to implement? As
> >> > first I
> >> > > > > > > suggested the master-slaves scenario I think it is easy to
> >> > > implement
> >> > > > > than
> >> > > > > > > distributed lock scenario. However if you can suggest one
> from
> >> > > these
> >> > > > > two,
> >> > > > > > > then I can think about it.
> >> > > > > > >
> >> > > > > > > Thank you
> >> > > > > > >
> >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com>
> >> > wrote:
> >> > > > > > >
> >> > > > > > > > With respect to the hotdeployment,
> >> > > > > > > >
> >> > > > > > > > We can drop the deployment archive onto the deployment
> >> folder.
> >> > > > Since
> >> > > > > > the
> >> > > > > > > > DeploymentPoller are acquiring the distributed lock for
> the
> >> > > > > > > DeploymentUnit,
> >> > > > > > > > only one of the nodes will get the lock and initiate the
> >> > > > deployment.
> >> > > > > > > > DeploymentPollers on other nodes will fail in acquiring
> the
> >> > lock
> >> > > > and
> >> > > > > > > hence
> >> > > > > > > > will silently ignore it.
> >> > > > > > > >
> >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> >> > > > sathwik.bp@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Tammo,
> >> > > > > > > > >
> >> > > > > > > > > The distributed lock acquisition on the DeploymentUnit
> >> should
> >> > > be
> >> > > > > > added
> >> > > > > > > to
> >> > > > > > > > > both DeploymentWebService and DeploymentPoller.
> >> > > > > > > > >
> >> > > > > > > > > When a deployment operation is initiated through the
> >> > > > > > > > DeploymentWebService,
> >> > > > > > > > > The load balancer routes it to any of the available
> nodes.
> >> > > > > > > > >
> >> > > > > > > > > On the routed node, the DeploymentWebService acquires
> the
> >> > > > > Distributed
> >> > > > > > > > > lock. On the remaining nodes the DeploymentPoller will
> >> try to
> >> > > > > acquire
> >> > > > > > > the
> >> > > > > > > > > distributed lock and will not get it and hence will
> >> silently
> >> > > > ignore
> >> > > > > > it.
> >> > > > > > > > >
> >> > > > > > > > > Once the routed node completes the deployment, it will
> >> > release
> >> > > > the
> >> > > > > > > lock.
> >> > > > > > > > > This way we don't have to stall the DeploymentPoller in
> >> other
> >> > > > > nodes.
> >> > > > > > > > >
> >> > > > > > > > > Does it answer the concerns?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Now, if we give the responsibility of identifying the
> >> master
> >> > > node
> >> > > > > to
> >> > > > > > > the
> >> > > > > > > > > hazelcast, how do we plan to intimate the load balancer
> to
> >> > > change
> >> > > > > > it's
> >> > > > > > > > > configuration about the master node?
> >> > > > > > > > > Assuming there are 3 nodes in the cluster,
> >> > > > > > > > > node1 -master
> >> > > > > > > > > node2 - slave
> >> > > > > > > > > node3 - slave
> >> > > > > > > > >
> >> > > > > > > > > Node1 goes down, the LB will promote Node2 as master
> node,
> >> > but
> >> > > > > > > hazelcast
> >> > > > > > > > > might promote Node3 as master node. They are out of
> sync.
> >> > > > > > > > >
> >> > > > > > > > > Is this argument valid?
> >> > > > > > > > >
> >> > > > > > > > > regards,
> >> > > > > > > > > sathwik
> >> > > > > > > > >
> >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> >> > > > > > > tvanlessen@gmail.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi Sudharma,
> >> > > > > > > > >>
> >> > > > > > > > >> what do you expect from the "other nodes deployment"?
> >> > > > Compilation
> >> > > > > is
> >> > > > > > > not
> >> > > > > > > > >> needed since the CBP file is written to the (shared)
> FS.
> >> > > > > > Registration
> >> > > > > > > is
> >> > > > > > > > >> also not needed, since it is done via the shared
> >> database.
> >> > So
> >> > > > the
> >> > > > > > only
> >> > > > > > > > >> thing that might be needed is to tell the engine that
> >> there
> >> > > is a
> >> > > > > new
> >> > > > > > > > >> deployment. I'd need to check that. If this is needed,
> I
> >> > > revert
> >> > > > my
> >> > > > > > > last
> >> > > > > > > > >> statement, then it is perhaps better to just send an
> >> event
> >> > > over
> >> > > > > > > > Hazelcast
> >> > > > > > > > >> to all nodes that the deployment has changed.
> >> > > > > > > > >>
> >> > > > > > > > >> Best,
> >> > > > > > > > >>   Tammo
> >> > > > > > > > >>
> >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> >> > > > > > > > >> suba.11@cse.mrt.ac.lk
> >> > > > > > > > >> > wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> > Hi Tammo,
> >> > > > > > > > >> >
> >> > > > > > > > >> > The master node writes meta data. But runtime
> >> information
> >> > > must
> >> > > > > be
> >> > > > > > > > >> available
> >> > > > > > > > >> > in all nodes.Since the folder is shared, all nodes
> will
> >> > see
> >> > > > the
> >> > > > > > > > >> > availability of a new process. My idea is for master
> >> node
> >> > to
> >> > > > > write
> >> > > > > > > the
> >> > > > > > > > >> meta
> >> > > > > > > > >> > data and other nodes to just read the meta data and
> >> load
> >> > > > > > process.So
> >> > > > > > > we
> >> > > > > > > > >> need
> >> > > > > > > > >> > a small delay between master node deployment and
> other
> >> > nodes
> >> > > > > > > > deployment.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Is there anyway to set the delay between master node
> >> and
> >> > > > slaves
> >> > > > > > > until
> >> > > > > > > > >> > master node finish the deployment?
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thank you
> >> > > > > > > > >> > Sudharma
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> >> > > > tvanlessen@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >> >
> >> > > > > > > > >> > > Hi Sathwik,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> >> > > > > > > sathwik.bp@gmail.com>
> >> > > > > > > > >> > wrote:
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > Sudharma/Tammo,
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > 1) How do we plan to decide which is the master
> >> node
> >> > in
> >> > > > the
> >> > > > > > > > cluster?
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I think the easiest approach is to always elect the
> >> > oldest
> >> > > > > node
> >> > > > > > in
> >> > > > > > > > the
> >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can
> easily
> >> > asked
> >> > > > for
> >> > > > > > > this
> >> > > > > > > > >> > > information.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > 2) Don't we need to stall the Deployment Pollers
> in
> >> > the
> >> > > > > slave
> >> > > > > > > > nodes?
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > Absolutely.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Suggestion:
> >> > > > > > > > >> > > > I am not sure whether do we need Master-SLaves.
> Why
> >> > not
> >> > > > give
> >> > > > > > > every
> >> > > > > > > > >> node
> >> > > > > > > > >> > > in
> >> > > > > > > > >> > > > the cluster the same status (Active-Active).
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > When a new deployment is made, the load balancer
> >> can
> >> > > push
> >> > > > it
> >> > > > > > to
> >> > > > > > > > any
> >> > > > > > > > >> of
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > available nodes. That node will probably acquire
> a
> >> > > > > distributed
> >> > > > > > > > lock
> >> > > > > > > > >> on
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > deployment unit and acts as master for that
> >> > deployment.
> >> > > > This
> >> > > > > > > > ensures
> >> > > > > > > > >> > > > optimum usage of the cluster nodes. Probably no
> >> static
> >> > > > > > > > >> configuration of
> >> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
> >> > hazelcast.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > But this would not allow to have the hotdeployment
> >> via
> >> > > > > > filesystem
> >> > > > > > > > >> still
> >> > > > > > > > >> > > enabled, right?
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Best,
> >> > > > > > > > >> > >   Tammo
> >> > > > > > > > >> > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > --
> >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >> --
> >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Sudharma,

Any updates?

regards,
sathwik

On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <sa...@gmail.com> wrote:

> Sudharma,
>
> Can you elaborate on your option 1).
>
> Response to your option 2).
>
>     Process Store is the component that handles process metadata,
> compilation and deployment in ODE. Integration layers in ODE (Axis2, JBI)
> use the process store.
>     Future implementations of IL for ODE will also use the process store.
> We should not be thinking of moving the process store functionality to the
> integration layers.
>
>
> On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk> wrote:
>
>> Hi,
>>
>> I understood the problem within dynamic master/slave configuration. In my
>> approach, when a deployment request is routed to a slave node there will
>> not be a deployment. I suggest two options to avoid it.
>> 1) Have static master/slave configuration only for deploy process
>>
> 2) Modify the deployment web service to complie and verify the process and
>> then copy it to the deploy folder irrespective of whether its a master or
>> slave, then deployment poller should take care of the deployment
>>
>>
>
>>
>> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
>>
>> > Sudharma,
>> >
>> > We definitely need a master/slave in the hazelcast cluster. This is
>> > probably needed for the job migration in the Scheduler to migrate the
>> jobs
>> > associated with a down node. Let hold on this topic for future
>> discussion.
>> >
>> > Going by the explanation where the master/slave nodes have certain
>> > predefined tasks to perform is perfectly fine.
>> >
>> > I have this scenario,
>> >
>> > I am using HAProxy as my load balancer and configured 3 nodes in the
>> > cluster.
>> >
>> > Node1 - Active
>> > Node2 - Active
>> > Node3 - Backup
>> >
>> > Load balancing algorithm: RoundRobin
>> >
>> > A Backup node (Node3) is one which the load balancer will not route
>> > requests to, until one of the Active node i.e either Node1 or Node2 has
>> > gone down.
>> >
>> > All these 3 nodes are also part of the hazelcast cluster as well.
>> >
>> > In the hazelcast cluster, assume Node1 is elected as the leader/master
>> and
>> > Node2,Node3 as slaves.
>> >
>> > I initiate the deploy operation on the DeploymentWebService which the
>> load
>> > balancer routes it to one of the Active nodes in the cluster, lets say
>> it's
>> > the Node1. Since Node1 is also the master in the hazelcast cluster,
>> > deployment is a success.
>> >
>> > I initiate another deploy operation on the DeploymentWebService which
>> the
>> > load balancer routes it to the next active node which is Node2. Since
>> Node2
>> > is a slave in the Hazelcast cluster, What happens to the deployment?
>> >
>> > regards,
>> > sathwik
>> >
>> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>> > suba.11@cse.mrt.ac.lk
>> > > wrote:
>> >
>> > > Hi,
>> > >
>> > > I will explain my approach as much as possible. The oldest node in the
>> > > hazelcast cluster is elected as the master node. In the failure of the
>> > > master node, next oldest node will be elected as the master node. This
>> > > master-slave configuration is just for deployment. When the hazelcast
>> > > cluster elected the master node, that node becomes a master node for
>> > > deploying process. So it will do the deploying artifacts. If you want
>> to
>> > > get the idea of electing master node please refer the code which I
>> have
>> > > located in the github. (
>> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>> > >
>> > > I identified separated actions which should be followed by the master
>> and
>> > > salve nodes.
>> > > Actions which are followed by master node only
>> > > 1) create deployment unit
>> > > 2) set the version nu to deployment unit
>> > > 3) compile deployment unit
>> > > 4) scan deployment unit
>> > > 5) retire previous versions
>> > > Master node and slave nodes should create _processes which stores
>> > > ProcessConfImpl
>> > > Only master node will write the version nu to database, create
>> .deployed
>> > > file
>> > >
>> > > So there are some actions which should be followed only by master node
>> > > while other actions should be followed by all the nodes.The idea of
>> > having
>> > > a master node is deploying artifacts and avoid others from writing the
>> > > version nu to database.
>> > > Whether a node is active or passive, all nodes should do the
>> > > deployment.Master
>> > > and slaves will follow necessary actions as in above.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com> wrote:
>> > >
>> > > > Nandika,
>> > > >
>> > > > I very well understand what you have put across, but it's secondary
>> to
>> > me
>> > > > now.
>> > > >
>> > > > Sudharma,
>> > > > My primary concern is to understand at a high level the deployment
>> > > > architecture and how would master-slave configuration fit in. Are
>> there
>> > > any
>> > > > restrictions imposed by the in-progress design?
>> > > >
>> > > > Firstly, how would ODE process deployment work under these cluster
>> > > > configurations?
>> > > >
>> > > > Sample Cluster configurations: A load balancer is frontending the
>> > > servers.
>> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>> > > > 3) Cluster with 2+ nodes with additional nodes either in Active or
>> > > Passive.
>> > > >
>> > > > regards,
>> > > > sathwik
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>> > jayawark@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Hi Sathwik,
>> > > > >
>> > > > > According to my understanding, in the clustering scenario, the
>> master
>> > > > node
>> > > > > should perform all the deployment actions and the slave nodes also
>> > need
>> > > > to
>> > > > > perform some deployment actions. For example, the slave nodes also
>> > > should
>> > > > > handle the process ACTIVATED event so that the process
>> configuration
>> > is
>> > > > > added to the engine and necessary web services are created so that
>> > when
>> > > > the
>> > > > > load balancer send requests to any node in the cluster, it is
>> ready
>> > to
>> > > > > accept those requests.
>> > > > >
>> > > > > Regards
>> > > > > Nandika
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>> sathwik.bp@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Sudharma,
>> > > > > >
>> > > > > > Where are you going to configure the master-slaves, is it in the
>> > web
>> > > > > > application level or at the load balancer?
>> > > > > >
>> > > > > > regards,
>> > > > > > sathwik
>> > > > > >
>> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
>> > > > > > suba.11@cse.mrt.ac.lk>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Tammo,
>> > > > > > >
>> > > > > > > Can you suggest the best method from these to implement? As
>> > first I
>> > > > > > > suggested the master-slaves scenario I think it is easy to
>> > > implement
>> > > > > than
>> > > > > > > distributed lock scenario. However if you can suggest one from
>> > > these
>> > > > > two,
>> > > > > > > then I can think about it.
>> > > > > > >
>> > > > > > > Thank you
>> > > > > > >
>> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com>
>> > wrote:
>> > > > > > >
>> > > > > > > > With respect to the hotdeployment,
>> > > > > > > >
>> > > > > > > > We can drop the deployment archive onto the deployment
>> folder.
>> > > > Since
>> > > > > > the
>> > > > > > > > DeploymentPoller are acquiring the distributed lock for the
>> > > > > > > DeploymentUnit,
>> > > > > > > > only one of the nodes will get the lock and initiate the
>> > > > deployment.
>> > > > > > > > DeploymentPollers on other nodes will fail in acquiring the
>> > lock
>> > > > and
>> > > > > > > hence
>> > > > > > > > will silently ignore it.
>> > > > > > > >
>> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>> > > > sathwik.bp@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Tammo,
>> > > > > > > > >
>> > > > > > > > > The distributed lock acquisition on the DeploymentUnit
>> should
>> > > be
>> > > > > > added
>> > > > > > > to
>> > > > > > > > > both DeploymentWebService and DeploymentPoller.
>> > > > > > > > >
>> > > > > > > > > When a deployment operation is initiated through the
>> > > > > > > > DeploymentWebService,
>> > > > > > > > > The load balancer routes it to any of the available nodes.
>> > > > > > > > >
>> > > > > > > > > On the routed node, the DeploymentWebService acquires the
>> > > > > Distributed
>> > > > > > > > > lock. On the remaining nodes the DeploymentPoller will
>> try to
>> > > > > acquire
>> > > > > > > the
>> > > > > > > > > distributed lock and will not get it and hence will
>> silently
>> > > > ignore
>> > > > > > it.
>> > > > > > > > >
>> > > > > > > > > Once the routed node completes the deployment, it will
>> > release
>> > > > the
>> > > > > > > lock.
>> > > > > > > > > This way we don't have to stall the DeploymentPoller in
>> other
>> > > > > nodes.
>> > > > > > > > >
>> > > > > > > > > Does it answer the concerns?
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Now, if we give the responsibility of identifying the
>> master
>> > > node
>> > > > > to
>> > > > > > > the
>> > > > > > > > > hazelcast, how do we plan to intimate the load balancer to
>> > > change
>> > > > > > it's
>> > > > > > > > > configuration about the master node?
>> > > > > > > > > Assuming there are 3 nodes in the cluster,
>> > > > > > > > > node1 -master
>> > > > > > > > > node2 - slave
>> > > > > > > > > node3 - slave
>> > > > > > > > >
>> > > > > > > > > Node1 goes down, the LB will promote Node2 as master node,
>> > but
>> > > > > > > hazelcast
>> > > > > > > > > might promote Node3 as master node. They are out of sync.
>> > > > > > > > >
>> > > > > > > > > Is this argument valid?
>> > > > > > > > >
>> > > > > > > > > regards,
>> > > > > > > > > sathwik
>> > > > > > > > >
>> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
>> > > > > > > tvanlessen@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi Sudharma,
>> > > > > > > > >>
>> > > > > > > > >> what do you expect from the "other nodes deployment"?
>> > > > Compilation
>> > > > > is
>> > > > > > > not
>> > > > > > > > >> needed since the CBP file is written to the (shared) FS.
>> > > > > > Registration
>> > > > > > > is
>> > > > > > > > >> also not needed, since it is done via the shared
>> database.
>> > So
>> > > > the
>> > > > > > only
>> > > > > > > > >> thing that might be needed is to tell the engine that
>> there
>> > > is a
>> > > > > new
>> > > > > > > > >> deployment. I'd need to check that. If this is needed, I
>> > > revert
>> > > > my
>> > > > > > > last
>> > > > > > > > >> statement, then it is perhaps better to just send an
>> event
>> > > over
>> > > > > > > > Hazelcast
>> > > > > > > > >> to all nodes that the deployment has changed.
>> > > > > > > > >>
>> > > > > > > > >> Best,
>> > > > > > > > >>   Tammo
>> > > > > > > > >>
>> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
>> > > > > > > > >> suba.11@cse.mrt.ac.lk
>> > > > > > > > >> > wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Hi Tammo,
>> > > > > > > > >> >
>> > > > > > > > >> > The master node writes meta data. But runtime
>> information
>> > > must
>> > > > > be
>> > > > > > > > >> available
>> > > > > > > > >> > in all nodes.Since the folder is shared, all nodes will
>> > see
>> > > > the
>> > > > > > > > >> > availability of a new process. My idea is for master
>> node
>> > to
>> > > > > write
>> > > > > > > the
>> > > > > > > > >> meta
>> > > > > > > > >> > data and other nodes to just read the meta data and
>> load
>> > > > > > process.So
>> > > > > > > we
>> > > > > > > > >> need
>> > > > > > > > >> > a small delay between master node deployment and other
>> > nodes
>> > > > > > > > deployment.
>> > > > > > > > >> >
>> > > > > > > > >> > Is there anyway to set the delay between master node
>> and
>> > > > slaves
>> > > > > > > until
>> > > > > > > > >> > master node finish the deployment?
>> > > > > > > > >> >
>> > > > > > > > >> > Thank you
>> > > > > > > > >> > Sudharma
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
>> > > > tvanlessen@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Hi Sathwik,
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
>> > > > > > > sathwik.bp@gmail.com>
>> > > > > > > > >> > wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > > > Sudharma/Tammo,
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > 1) How do we plan to decide which is the master
>> node
>> > in
>> > > > the
>> > > > > > > > cluster?
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > I think the easiest approach is to always elect the
>> > oldest
>> > > > > node
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can easily
>> > asked
>> > > > for
>> > > > > > > this
>> > > > > > > > >> > > information.
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > > 2) Don't we need to stall the Deployment Pollers in
>> > the
>> > > > > slave
>> > > > > > > > nodes?
>> > > > > > > > >> > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > > Absolutely.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Suggestion:
>> > > > > > > > >> > > > I am not sure whether do we need Master-SLaves. Why
>> > not
>> > > > give
>> > > > > > > every
>> > > > > > > > >> node
>> > > > > > > > >> > > in
>> > > > > > > > >> > > > the cluster the same status (Active-Active).
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > When a new deployment is made, the load balancer
>> can
>> > > push
>> > > > it
>> > > > > > to
>> > > > > > > > any
>> > > > > > > > >> of
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > available nodes. That node will probably acquire a
>> > > > > distributed
>> > > > > > > > lock
>> > > > > > > > >> on
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > deployment unit and acts as master for that
>> > deployment.
>> > > > This
>> > > > > > > > ensures
>> > > > > > > > >> > > > optimum usage of the cluster nodes. Probably no
>> static
>> > > > > > > > >> configuration of
>> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
>> > hazelcast.
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > But this would not allow to have the hotdeployment
>> via
>> > > > > > filesystem
>> > > > > > > > >> still
>> > > > > > > > >> > > enabled, right?
>> > > > > > > > >> > >
>> > > > > > > > >> > > Best,
>> > > > > > > > >> > >   Tammo
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > --
>> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> --
>> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Sudharma,

Can you elaborate on your option 1).

Response to your option 2).

    Process Store is the component that handles process metadata,
compilation and deployment in ODE. Integration layers in ODE (Axis2, JBI)
use the process store.
    Future implementations of IL for ODE will also use the process store.
We should not be thinking of moving the process store functionality to the
integration layers.

On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <su...@cse.mrt.ac.lk>
wrote:

> Hi,
>
> I understood the problem within dynamic master/slave configuration. In my
> approach, when a deployment request is routed to a slave node there will
> not be a deployment. I suggest two options to avoid it.
> 1) Have static master/slave configuration only for deploy process
>
2) Modify the deployment web service to complie and verify the process and
> then copy it to the deploy folder irrespective of whether its a master or
> slave, then deployment poller should take care of the deployment
>
>

>
> On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:
>
> > Sudharma,
> >
> > We definitely need a master/slave in the hazelcast cluster. This is
> > probably needed for the job migration in the Scheduler to migrate the
> jobs
> > associated with a down node. Let hold on this topic for future
> discussion.
> >
> > Going by the explanation where the master/slave nodes have certain
> > predefined tasks to perform is perfectly fine.
> >
> > I have this scenario,
> >
> > I am using HAProxy as my load balancer and configured 3 nodes in the
> > cluster.
> >
> > Node1 - Active
> > Node2 - Active
> > Node3 - Backup
> >
> > Load balancing algorithm: RoundRobin
> >
> > A Backup node (Node3) is one which the load balancer will not route
> > requests to, until one of the Active node i.e either Node1 or Node2 has
> > gone down.
> >
> > All these 3 nodes are also part of the hazelcast cluster as well.
> >
> > In the hazelcast cluster, assume Node1 is elected as the leader/master
> and
> > Node2,Node3 as slaves.
> >
> > I initiate the deploy operation on the DeploymentWebService which the
> load
> > balancer routes it to one of the Active nodes in the cluster, lets say
> it's
> > the Node1. Since Node1 is also the master in the hazelcast cluster,
> > deployment is a success.
> >
> > I initiate another deploy operation on the DeploymentWebService which the
> > load balancer routes it to the next active node which is Node2. Since
> Node2
> > is a slave in the Hazelcast cluster, What happens to the deployment?
> >
> > regards,
> > sathwik
> >
> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> > suba.11@cse.mrt.ac.lk
> > > wrote:
> >
> > > Hi,
> > >
> > > I will explain my approach as much as possible. The oldest node in the
> > > hazelcast cluster is elected as the master node. In the failure of the
> > > master node, next oldest node will be elected as the master node. This
> > > master-slave configuration is just for deployment. When the hazelcast
> > > cluster elected the master node, that node becomes a master node for
> > > deploying process. So it will do the deploying artifacts. If you want
> to
> > > get the idea of electing master node please refer the code which I have
> > > located in the github. (
> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
> > >
> > > I identified separated actions which should be followed by the master
> and
> > > salve nodes.
> > > Actions which are followed by master node only
> > > 1) create deployment unit
> > > 2) set the version nu to deployment unit
> > > 3) compile deployment unit
> > > 4) scan deployment unit
> > > 5) retire previous versions
> > > Master node and slave nodes should create _processes which stores
> > > ProcessConfImpl
> > > Only master node will write the version nu to database, create
> .deployed
> > > file
> > >
> > > So there are some actions which should be followed only by master node
> > > while other actions should be followed by all the nodes.The idea of
> > having
> > > a master node is deploying artifacts and avoid others from writing the
> > > version nu to database.
> > > Whether a node is active or passive, all nodes should do the
> > > deployment.Master
> > > and slaves will follow necessary actions as in above.
> > >
> > >
> > >
> > >
> > >
> > > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com> wrote:
> > >
> > > > Nandika,
> > > >
> > > > I very well understand what you have put across, but it's secondary
> to
> > me
> > > > now.
> > > >
> > > > Sudharma,
> > > > My primary concern is to understand at a high level the deployment
> > > > architecture and how would master-slave configuration fit in. Are
> there
> > > any
> > > > restrictions imposed by the in-progress design?
> > > >
> > > > Firstly, how would ODE process deployment work under these cluster
> > > > configurations?
> > > >
> > > > Sample Cluster configurations: A load balancer is frontending the
> > > servers.
> > > > 1) Cluster consisting of 2 nodes all Active-Active.
> > > > 2) Cluster consisting of 2 nodes Active-Passive.
> > > > 3) Cluster with 2+ nodes with additional nodes either in Active or
> > > Passive.
> > > >
> > > > regards,
> > > > sathwik
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> > jayawark@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Sathwik,
> > > > >
> > > > > According to my understanding, in the clustering scenario, the
> master
> > > > node
> > > > > should perform all the deployment actions and the slave nodes also
> > need
> > > > to
> > > > > perform some deployment actions. For example, the slave nodes also
> > > should
> > > > > handle the process ACTIVATED event so that the process
> configuration
> > is
> > > > > added to the engine and necessary web services are created so that
> > when
> > > > the
> > > > > load balancer send requests to any node in the cluster, it is ready
> > to
> > > > > accept those requests.
> > > > >
> > > > > Regards
> > > > > Nandika
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
> sathwik.bp@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Sudharma,
> > > > > >
> > > > > > Where are you going to configure the master-slaves, is it in the
> > web
> > > > > > application level or at the load balancer?
> > > > > >
> > > > > > regards,
> > > > > > sathwik
> > > > > >
> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > > > > > suba.11@cse.mrt.ac.lk>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Tammo,
> > > > > > >
> > > > > > > Can you suggest the best method from these to implement? As
> > first I
> > > > > > > suggested the master-slaves scenario I think it is easy to
> > > implement
> > > > > than
> > > > > > > distributed lock scenario. However if you can suggest one from
> > > these
> > > > > two,
> > > > > > > then I can think about it.
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > > With respect to the hotdeployment,
> > > > > > > >
> > > > > > > > We can drop the deployment archive onto the deployment
> folder.
> > > > Since
> > > > > > the
> > > > > > > > DeploymentPoller are acquiring the distributed lock for the
> > > > > > > DeploymentUnit,
> > > > > > > > only one of the nodes will get the lock and initiate the
> > > > deployment.
> > > > > > > > DeploymentPollers on other nodes will fail in acquiring the
> > lock
> > > > and
> > > > > > > hence
> > > > > > > > will silently ignore it.
> > > > > > > >
> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> > > > sathwik.bp@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Tammo,
> > > > > > > > >
> > > > > > > > > The distributed lock acquisition on the DeploymentUnit
> should
> > > be
> > > > > > added
> > > > > > > to
> > > > > > > > > both DeploymentWebService and DeploymentPoller.
> > > > > > > > >
> > > > > > > > > When a deployment operation is initiated through the
> > > > > > > > DeploymentWebService,
> > > > > > > > > The load balancer routes it to any of the available nodes.
> > > > > > > > >
> > > > > > > > > On the routed node, the DeploymentWebService acquires the
> > > > > Distributed
> > > > > > > > > lock. On the remaining nodes the DeploymentPoller will try
> to
> > > > > acquire
> > > > > > > the
> > > > > > > > > distributed lock and will not get it and hence will
> silently
> > > > ignore
> > > > > > it.
> > > > > > > > >
> > > > > > > > > Once the routed node completes the deployment, it will
> > release
> > > > the
> > > > > > > lock.
> > > > > > > > > This way we don't have to stall the DeploymentPoller in
> other
> > > > > nodes.
> > > > > > > > >
> > > > > > > > > Does it answer the concerns?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now, if we give the responsibility of identifying the
> master
> > > node
> > > > > to
> > > > > > > the
> > > > > > > > > hazelcast, how do we plan to intimate the load balancer to
> > > change
> > > > > > it's
> > > > > > > > > configuration about the master node?
> > > > > > > > > Assuming there are 3 nodes in the cluster,
> > > > > > > > > node1 -master
> > > > > > > > > node2 - slave
> > > > > > > > > node3 - slave
> > > > > > > > >
> > > > > > > > > Node1 goes down, the LB will promote Node2 as master node,
> > but
> > > > > > > hazelcast
> > > > > > > > > might promote Node3 as master node. They are out of sync.
> > > > > > > > >
> > > > > > > > > Is this argument valid?
> > > > > > > > >
> > > > > > > > > regards,
> > > > > > > > > sathwik
> > > > > > > > >
> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > > > > > > tvanlessen@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Sudharma,
> > > > > > > > >>
> > > > > > > > >> what do you expect from the "other nodes deployment"?
> > > > Compilation
> > > > > is
> > > > > > > not
> > > > > > > > >> needed since the CBP file is written to the (shared) FS.
> > > > > > Registration
> > > > > > > is
> > > > > > > > >> also not needed, since it is done via the shared database.
> > So
> > > > the
> > > > > > only
> > > > > > > > >> thing that might be needed is to tell the engine that
> there
> > > is a
> > > > > new
> > > > > > > > >> deployment. I'd need to check that. If this is needed, I
> > > revert
> > > > my
> > > > > > > last
> > > > > > > > >> statement, then it is perhaps better to just send an event
> > > over
> > > > > > > > Hazelcast
> > > > > > > > >> to all nodes that the deployment has changed.
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >>   Tammo
> > > > > > > > >>
> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > > > > > > > >> suba.11@cse.mrt.ac.lk
> > > > > > > > >> > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Tammo,
> > > > > > > > >> >
> > > > > > > > >> > The master node writes meta data. But runtime
> information
> > > must
> > > > > be
> > > > > > > > >> available
> > > > > > > > >> > in all nodes.Since the folder is shared, all nodes will
> > see
> > > > the
> > > > > > > > >> > availability of a new process. My idea is for master
> node
> > to
> > > > > write
> > > > > > > the
> > > > > > > > >> meta
> > > > > > > > >> > data and other nodes to just read the meta data and load
> > > > > > process.So
> > > > > > > we
> > > > > > > > >> need
> > > > > > > > >> > a small delay between master node deployment and other
> > nodes
> > > > > > > > deployment.
> > > > > > > > >> >
> > > > > > > > >> > Is there anyway to set the delay between master node and
> > > > slaves
> > > > > > > until
> > > > > > > > >> > master node finish the deployment?
> > > > > > > > >> >
> > > > > > > > >> > Thank you
> > > > > > > > >> > Sudharma
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> > > > tvanlessen@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Sathwik,
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > > > > > > sathwik.bp@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Sudharma/Tammo,
> > > > > > > > >> > > >
> > > > > > > > >> > > > 1) How do we plan to decide which is the master node
> > in
> > > > the
> > > > > > > > cluster?
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > > I think the easiest approach is to always elect the
> > oldest
> > > > > node
> > > > > > in
> > > > > > > > the
> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can easily
> > asked
> > > > for
> > > > > > > this
> > > > > > > > >> > > information.
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > > 2) Don't we need to stall the Deployment Pollers in
> > the
> > > > > slave
> > > > > > > > nodes?
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > Absolutely.
> > > > > > > > >> > >
> > > > > > > > >> > > Suggestion:
> > > > > > > > >> > > > I am not sure whether do we need Master-SLaves. Why
> > not
> > > > give
> > > > > > > every
> > > > > > > > >> node
> > > > > > > > >> > > in
> > > > > > > > >> > > > the cluster the same status (Active-Active).
> > > > > > > > >> > > >
> > > > > > > > >> > > > When a new deployment is made, the load balancer can
> > > push
> > > > it
> > > > > > to
> > > > > > > > any
> > > > > > > > >> of
> > > > > > > > >> > > the
> > > > > > > > >> > > > available nodes. That node will probably acquire a
> > > > > distributed
> > > > > > > > lock
> > > > > > > > >> on
> > > > > > > > >> > > the
> > > > > > > > >> > > > deployment unit and acts as master for that
> > deployment.
> > > > This
> > > > > > > > ensures
> > > > > > > > >> > > > optimum usage of the cluster nodes. Probably no
> static
> > > > > > > > >> configuration of
> > > > > > > > >> > > > Master-Slave in the load balancer nor in the
> > hazelcast.
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > > But this would not allow to have the hotdeployment via
> > > > > > filesystem
> > > > > > > > >> still
> > > > > > > > >> > > enabled, right?
> > > > > > > > >> > >
> > > > > > > > >> > > Best,
> > > > > > > > >> > >   Tammo
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > --
> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Tammo van Lessen - http://www.taval.de
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi,

I understood the problem within dynamic master/slave configuration. In my
approach, when a deployment request is routed to a slave node there will
not be a deployment. I suggest two options to avoid it.
1) Have static master/slave configuration only for deploy process
2) Modify the deployment web service to complie and verify the process and
then copy it to the deploy folder irrespective of whether its a master or
slave, then deployment poller should take care of the deployment


On 28 May 2015 at 14:43, Sathwik B P <sa...@gmail.com> wrote:

> Sudharma,
>
> We definitely need a master/slave in the hazelcast cluster. This is
> probably needed for the job migration in the Scheduler to migrate the jobs
> associated with a down node. Let hold on this topic for future discussion.
>
> Going by the explanation where the master/slave nodes have certain
> predefined tasks to perform is perfectly fine.
>
> I have this scenario,
>
> I am using HAProxy as my load balancer and configured 3 nodes in the
> cluster.
>
> Node1 - Active
> Node2 - Active
> Node3 - Backup
>
> Load balancing algorithm: RoundRobin
>
> A Backup node (Node3) is one which the load balancer will not route
> requests to, until one of the Active node i.e either Node1 or Node2 has
> gone down.
>
> All these 3 nodes are also part of the hazelcast cluster as well.
>
> In the hazelcast cluster, assume Node1 is elected as the leader/master and
> Node2,Node3 as slaves.
>
> I initiate the deploy operation on the DeploymentWebService which the load
> balancer routes it to one of the Active nodes in the cluster, lets say it's
> the Node1. Since Node1 is also the master in the hazelcast cluster,
> deployment is a success.
>
> I initiate another deploy operation on the DeploymentWebService which the
> load balancer routes it to the next active node which is Node2. Since Node2
> is a slave in the Hazelcast cluster, What happens to the deployment?
>
> regards,
> sathwik
>
> On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk
> > wrote:
>
> > Hi,
> >
> > I will explain my approach as much as possible. The oldest node in the
> > hazelcast cluster is elected as the master node. In the failure of the
> > master node, next oldest node will be elected as the master node. This
> > master-slave configuration is just for deployment. When the hazelcast
> > cluster elected the master node, that node becomes a master node for
> > deploying process. So it will do the deploying artifacts. If you want to
> > get the idea of electing master node please refer the code which I have
> > located in the github. (
> > https://github.com/Subasinghe/ode/tree/ode_clustering)
> >
> > I identified separated actions which should be followed by the master and
> > salve nodes.
> > Actions which are followed by master node only
> > 1) create deployment unit
> > 2) set the version nu to deployment unit
> > 3) compile deployment unit
> > 4) scan deployment unit
> > 5) retire previous versions
> > Master node and slave nodes should create _processes which stores
> > ProcessConfImpl
> > Only master node will write the version nu to database, create .deployed
> > file
> >
> > So there are some actions which should be followed only by master node
> > while other actions should be followed by all the nodes.The idea of
> having
> > a master node is deploying artifacts and avoid others from writing the
> > version nu to database.
> > Whether a node is active or passive, all nodes should do the
> > deployment.Master
> > and slaves will follow necessary actions as in above.
> >
> >
> >
> >
> >
> > On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com> wrote:
> >
> > > Nandika,
> > >
> > > I very well understand what you have put across, but it's secondary to
> me
> > > now.
> > >
> > > Sudharma,
> > > My primary concern is to understand at a high level the deployment
> > > architecture and how would master-slave configuration fit in. Are there
> > any
> > > restrictions imposed by the in-progress design?
> > >
> > > Firstly, how would ODE process deployment work under these cluster
> > > configurations?
> > >
> > > Sample Cluster configurations: A load balancer is frontending the
> > servers.
> > > 1) Cluster consisting of 2 nodes all Active-Active.
> > > 2) Cluster consisting of 2 nodes Active-Passive.
> > > 3) Cluster with 2+ nodes with additional nodes either in Active or
> > Passive.
> > >
> > > regards,
> > > sathwik
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> jayawark@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Sathwik,
> > > >
> > > > According to my understanding, in the clustering scenario, the master
> > > node
> > > > should perform all the deployment actions and the slave nodes also
> need
> > > to
> > > > perform some deployment actions. For example, the slave nodes also
> > should
> > > > handle the process ACTIVATED event so that the process configuration
> is
> > > > added to the engine and necessary web services are created so that
> when
> > > the
> > > > load balancer send requests to any node in the cluster, it is ready
> to
> > > > accept those requests.
> > > >
> > > > Regards
> > > > Nandika
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Sudharma,
> > > > >
> > > > > Where are you going to configure the master-slaves, is it in the
> web
> > > > > application level or at the load balancer?
> > > > >
> > > > > regards,
> > > > > sathwik
> > > > >
> > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > > > > suba.11@cse.mrt.ac.lk>
> > > > > wrote:
> > > > >
> > > > > > Hi Tammo,
> > > > > >
> > > > > > Can you suggest the best method from these to implement? As
> first I
> > > > > > suggested the master-slaves scenario I think it is easy to
> > implement
> > > > than
> > > > > > distributed lock scenario. However if you can suggest one from
> > these
> > > > two,
> > > > > > then I can think about it.
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com>
> wrote:
> > > > > >
> > > > > > > With respect to the hotdeployment,
> > > > > > >
> > > > > > > We can drop the deployment archive onto the deployment folder.
> > > Since
> > > > > the
> > > > > > > DeploymentPoller are acquiring the distributed lock for the
> > > > > > DeploymentUnit,
> > > > > > > only one of the nodes will get the lock and initiate the
> > > deployment.
> > > > > > > DeploymentPollers on other nodes will fail in acquiring the
> lock
> > > and
> > > > > > hence
> > > > > > > will silently ignore it.
> > > > > > >
> > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> > > sathwik.bp@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tammo,
> > > > > > > >
> > > > > > > > The distributed lock acquisition on the DeploymentUnit should
> > be
> > > > > added
> > > > > > to
> > > > > > > > both DeploymentWebService and DeploymentPoller.
> > > > > > > >
> > > > > > > > When a deployment operation is initiated through the
> > > > > > > DeploymentWebService,
> > > > > > > > The load balancer routes it to any of the available nodes.
> > > > > > > >
> > > > > > > > On the routed node, the DeploymentWebService acquires the
> > > > Distributed
> > > > > > > > lock. On the remaining nodes the DeploymentPoller will try to
> > > > acquire
> > > > > > the
> > > > > > > > distributed lock and will not get it and hence will silently
> > > ignore
> > > > > it.
> > > > > > > >
> > > > > > > > Once the routed node completes the deployment, it will
> release
> > > the
> > > > > > lock.
> > > > > > > > This way we don't have to stall the DeploymentPoller in other
> > > > nodes.
> > > > > > > >
> > > > > > > > Does it answer the concerns?
> > > > > > > >
> > > > > > > >
> > > > > > > > Now, if we give the responsibility of identifying the master
> > node
> > > > to
> > > > > > the
> > > > > > > > hazelcast, how do we plan to intimate the load balancer to
> > change
> > > > > it's
> > > > > > > > configuration about the master node?
> > > > > > > > Assuming there are 3 nodes in the cluster,
> > > > > > > > node1 -master
> > > > > > > > node2 - slave
> > > > > > > > node3 - slave
> > > > > > > >
> > > > > > > > Node1 goes down, the LB will promote Node2 as master node,
> but
> > > > > > hazelcast
> > > > > > > > might promote Node3 as master node. They are out of sync.
> > > > > > > >
> > > > > > > > Is this argument valid?
> > > > > > > >
> > > > > > > > regards,
> > > > > > > > sathwik
> > > > > > > >
> > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > > > > > tvanlessen@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Sudharma,
> > > > > > > >>
> > > > > > > >> what do you expect from the "other nodes deployment"?
> > > Compilation
> > > > is
> > > > > > not
> > > > > > > >> needed since the CBP file is written to the (shared) FS.
> > > > > Registration
> > > > > > is
> > > > > > > >> also not needed, since it is done via the shared database.
> So
> > > the
> > > > > only
> > > > > > > >> thing that might be needed is to tell the engine that there
> > is a
> > > > new
> > > > > > > >> deployment. I'd need to check that. If this is needed, I
> > revert
> > > my
> > > > > > last
> > > > > > > >> statement, then it is perhaps better to just send an event
> > over
> > > > > > > Hazelcast
> > > > > > > >> to all nodes that the deployment has changed.
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >>   Tammo
> > > > > > > >>
> > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > > > > > > >> suba.11@cse.mrt.ac.lk
> > > > > > > >> > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Tammo,
> > > > > > > >> >
> > > > > > > >> > The master node writes meta data. But runtime information
> > must
> > > > be
> > > > > > > >> available
> > > > > > > >> > in all nodes.Since the folder is shared, all nodes will
> see
> > > the
> > > > > > > >> > availability of a new process. My idea is for master node
> to
> > > > write
> > > > > > the
> > > > > > > >> meta
> > > > > > > >> > data and other nodes to just read the meta data and load
> > > > > process.So
> > > > > > we
> > > > > > > >> need
> > > > > > > >> > a small delay between master node deployment and other
> nodes
> > > > > > > deployment.
> > > > > > > >> >
> > > > > > > >> > Is there anyway to set the delay between master node and
> > > slaves
> > > > > > until
> > > > > > > >> > master node finish the deployment?
> > > > > > > >> >
> > > > > > > >> > Thank you
> > > > > > > >> > Sudharma
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> > > tvanlessen@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Sathwik,
> > > > > > > >> > >
> > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > > > > > sathwik.bp@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Sudharma/Tammo,
> > > > > > > >> > > >
> > > > > > > >> > > > 1) How do we plan to decide which is the master node
> in
> > > the
> > > > > > > cluster?
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > > I think the easiest approach is to always elect the
> oldest
> > > > node
> > > > > in
> > > > > > > the
> > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can easily
> asked
> > > for
> > > > > > this
> > > > > > > >> > > information.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > > 2) Don't we need to stall the Deployment Pollers in
> the
> > > > slave
> > > > > > > nodes?
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > Absolutely.
> > > > > > > >> > >
> > > > > > > >> > > Suggestion:
> > > > > > > >> > > > I am not sure whether do we need Master-SLaves. Why
> not
> > > give
> > > > > > every
> > > > > > > >> node
> > > > > > > >> > > in
> > > > > > > >> > > > the cluster the same status (Active-Active).
> > > > > > > >> > > >
> > > > > > > >> > > > When a new deployment is made, the load balancer can
> > push
> > > it
> > > > > to
> > > > > > > any
> > > > > > > >> of
> > > > > > > >> > > the
> > > > > > > >> > > > available nodes. That node will probably acquire a
> > > > distributed
> > > > > > > lock
> > > > > > > >> on
> > > > > > > >> > > the
> > > > > > > >> > > > deployment unit and acts as master for that
> deployment.
> > > This
> > > > > > > ensures
> > > > > > > >> > > > optimum usage of the cluster nodes. Probably no static
> > > > > > > >> configuration of
> > > > > > > >> > > > Master-Slave in the load balancer nor in the
> hazelcast.
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > > But this would not allow to have the hotdeployment via
> > > > > filesystem
> > > > > > > >> still
> > > > > > > >> > > enabled, right?
> > > > > > > >> > >
> > > > > > > >> > > Best,
> > > > > > > >> > >   Tammo
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Tammo van Lessen - http://www.taval.de
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Sudharma,

We definitely need a master/slave in the hazelcast cluster. This is
probably needed for the job migration in the Scheduler to migrate the jobs
associated with a down node. Let hold on this topic for future discussion.

Going by the explanation where the master/slave nodes have certain
predefined tasks to perform is perfectly fine.

I have this scenario,

I am using HAProxy as my load balancer and configured 3 nodes in the
cluster.

Node1 - Active
Node2 - Active
Node3 - Backup

Load balancing algorithm: RoundRobin

A Backup node (Node3) is one which the load balancer will not route
requests to, until one of the Active node i.e either Node1 or Node2 has
gone down.

All these 3 nodes are also part of the hazelcast cluster as well.

In the hazelcast cluster, assume Node1 is elected as the leader/master and
Node2,Node3 as slaves.

I initiate the deploy operation on the DeploymentWebService which the load
balancer routes it to one of the Active nodes in the cluster, lets say it's
the Node1. Since Node1 is also the master in the hazelcast cluster,
deployment is a success.

I initiate another deploy operation on the DeploymentWebService which the
load balancer routes it to the next active node which is Node2. Since Node2
is a slave in the Hazelcast cluster, What happens to the deployment?

regards,
sathwik

On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <suba.11@cse.mrt.ac.lk
> wrote:

> Hi,
>
> I will explain my approach as much as possible. The oldest node in the
> hazelcast cluster is elected as the master node. In the failure of the
> master node, next oldest node will be elected as the master node. This
> master-slave configuration is just for deployment. When the hazelcast
> cluster elected the master node, that node becomes a master node for
> deploying process. So it will do the deploying artifacts. If you want to
> get the idea of electing master node please refer the code which I have
> located in the github. (
> https://github.com/Subasinghe/ode/tree/ode_clustering)
>
> I identified separated actions which should be followed by the master and
> salve nodes.
> Actions which are followed by master node only
> 1) create deployment unit
> 2) set the version nu to deployment unit
> 3) compile deployment unit
> 4) scan deployment unit
> 5) retire previous versions
> Master node and slave nodes should create _processes which stores
> ProcessConfImpl
> Only master node will write the version nu to database, create .deployed
> file
>
> So there are some actions which should be followed only by master node
> while other actions should be followed by all the nodes.The idea of having
> a master node is deploying artifacts and avoid others from writing the
> version nu to database.
> Whether a node is active or passive, all nodes should do the
> deployment.Master
> and slaves will follow necessary actions as in above.
>
>
>
>
>
> On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com> wrote:
>
> > Nandika,
> >
> > I very well understand what you have put across, but it's secondary to me
> > now.
> >
> > Sudharma,
> > My primary concern is to understand at a high level the deployment
> > architecture and how would master-slave configuration fit in. Are there
> any
> > restrictions imposed by the in-progress design?
> >
> > Firstly, how would ODE process deployment work under these cluster
> > configurations?
> >
> > Sample Cluster configurations: A load balancer is frontending the
> servers.
> > 1) Cluster consisting of 2 nodes all Active-Active.
> > 2) Cluster consisting of 2 nodes Active-Passive.
> > 3) Cluster with 2+ nodes with additional nodes either in Active or
> Passive.
> >
> > regards,
> > sathwik
> >
> >
> >
> >
> >
> >
> >
> > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <jayawark@gmail.com
> >
> > wrote:
> >
> > > Hi Sathwik,
> > >
> > > According to my understanding, in the clustering scenario, the master
> > node
> > > should perform all the deployment actions and the slave nodes also need
> > to
> > > perform some deployment actions. For example, the slave nodes also
> should
> > > handle the process ACTIVATED event so that the process configuration is
> > > added to the engine and necessary web services are created so that when
> > the
> > > load balancer send requests to any node in the cluster, it is ready to
> > > accept those requests.
> > >
> > > Regards
> > > Nandika
> > >
> > >
> > >
> > >
> > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> > > wrote:
> > >
> > > > Sudharma,
> > > >
> > > > Where are you going to configure the master-slaves, is it in the web
> > > > application level or at the load balancer?
> > > >
> > > > regards,
> > > > sathwik
> > > >
> > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > > > suba.11@cse.mrt.ac.lk>
> > > > wrote:
> > > >
> > > > > Hi Tammo,
> > > > >
> > > > > Can you suggest the best method from these to implement? As first I
> > > > > suggested the master-slaves scenario I think it is easy to
> implement
> > > than
> > > > > distributed lock scenario. However if you can suggest one from
> these
> > > two,
> > > > > then I can think about it.
> > > > >
> > > > > Thank you
> > > > >
> > > > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com> wrote:
> > > > >
> > > > > > With respect to the hotdeployment,
> > > > > >
> > > > > > We can drop the deployment archive onto the deployment folder.
> > Since
> > > > the
> > > > > > DeploymentPoller are acquiring the distributed lock for the
> > > > > DeploymentUnit,
> > > > > > only one of the nodes will get the lock and initiate the
> > deployment.
> > > > > > DeploymentPollers on other nodes will fail in acquiring the lock
> > and
> > > > > hence
> > > > > > will silently ignore it.
> > > > > >
> > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> > sathwik.bp@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Tammo,
> > > > > > >
> > > > > > > The distributed lock acquisition on the DeploymentUnit should
> be
> > > > added
> > > > > to
> > > > > > > both DeploymentWebService and DeploymentPoller.
> > > > > > >
> > > > > > > When a deployment operation is initiated through the
> > > > > > DeploymentWebService,
> > > > > > > The load balancer routes it to any of the available nodes.
> > > > > > >
> > > > > > > On the routed node, the DeploymentWebService acquires the
> > > Distributed
> > > > > > > lock. On the remaining nodes the DeploymentPoller will try to
> > > acquire
> > > > > the
> > > > > > > distributed lock and will not get it and hence will silently
> > ignore
> > > > it.
> > > > > > >
> > > > > > > Once the routed node completes the deployment, it will release
> > the
> > > > > lock.
> > > > > > > This way we don't have to stall the DeploymentPoller in other
> > > nodes.
> > > > > > >
> > > > > > > Does it answer the concerns?
> > > > > > >
> > > > > > >
> > > > > > > Now, if we give the responsibility of identifying the master
> node
> > > to
> > > > > the
> > > > > > > hazelcast, how do we plan to intimate the load balancer to
> change
> > > > it's
> > > > > > > configuration about the master node?
> > > > > > > Assuming there are 3 nodes in the cluster,
> > > > > > > node1 -master
> > > > > > > node2 - slave
> > > > > > > node3 - slave
> > > > > > >
> > > > > > > Node1 goes down, the LB will promote Node2 as master node, but
> > > > > hazelcast
> > > > > > > might promote Node3 as master node. They are out of sync.
> > > > > > >
> > > > > > > Is this argument valid?
> > > > > > >
> > > > > > > regards,
> > > > > > > sathwik
> > > > > > >
> > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > > > > tvanlessen@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Sudharma,
> > > > > > >>
> > > > > > >> what do you expect from the "other nodes deployment"?
> > Compilation
> > > is
> > > > > not
> > > > > > >> needed since the CBP file is written to the (shared) FS.
> > > > Registration
> > > > > is
> > > > > > >> also not needed, since it is done via the shared database. So
> > the
> > > > only
> > > > > > >> thing that might be needed is to tell the engine that there
> is a
> > > new
> > > > > > >> deployment. I'd need to check that. If this is needed, I
> revert
> > my
> > > > > last
> > > > > > >> statement, then it is perhaps better to just send an event
> over
> > > > > > Hazelcast
> > > > > > >> to all nodes that the deployment has changed.
> > > > > > >>
> > > > > > >> Best,
> > > > > > >>   Tammo
> > > > > > >>
> > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > > > > > >> suba.11@cse.mrt.ac.lk
> > > > > > >> > wrote:
> > > > > > >>
> > > > > > >> > Hi Tammo,
> > > > > > >> >
> > > > > > >> > The master node writes meta data. But runtime information
> must
> > > be
> > > > > > >> available
> > > > > > >> > in all nodes.Since the folder is shared, all nodes will see
> > the
> > > > > > >> > availability of a new process. My idea is for master node to
> > > write
> > > > > the
> > > > > > >> meta
> > > > > > >> > data and other nodes to just read the meta data and load
> > > > process.So
> > > > > we
> > > > > > >> need
> > > > > > >> > a small delay between master node deployment and other nodes
> > > > > > deployment.
> > > > > > >> >
> > > > > > >> > Is there anyway to set the delay between master node and
> > slaves
> > > > > until
> > > > > > >> > master node finish the deployment?
> > > > > > >> >
> > > > > > >> > Thank you
> > > > > > >> > Sudharma
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> > tvanlessen@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Sathwik,
> > > > > > >> > >
> > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > > > > sathwik.bp@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > > > Sudharma/Tammo,
> > > > > > >> > > >
> > > > > > >> > > > 1) How do we plan to decide which is the master node in
> > the
> > > > > > cluster?
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > > I think the easiest approach is to always elect the oldest
> > > node
> > > > in
> > > > > > the
> > > > > > >> > > cluster to be the master. AFAIK Hazelcast can easily asked
> > for
> > > > > this
> > > > > > >> > > information.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > > 2) Don't we need to stall the Deployment Pollers in the
> > > slave
> > > > > > nodes?
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > Absolutely.
> > > > > > >> > >
> > > > > > >> > > Suggestion:
> > > > > > >> > > > I am not sure whether do we need Master-SLaves. Why not
> > give
> > > > > every
> > > > > > >> node
> > > > > > >> > > in
> > > > > > >> > > > the cluster the same status (Active-Active).
> > > > > > >> > > >
> > > > > > >> > > > When a new deployment is made, the load balancer can
> push
> > it
> > > > to
> > > > > > any
> > > > > > >> of
> > > > > > >> > > the
> > > > > > >> > > > available nodes. That node will probably acquire a
> > > distributed
> > > > > > lock
> > > > > > >> on
> > > > > > >> > > the
> > > > > > >> > > > deployment unit and acts as master for that deployment.
> > This
> > > > > > ensures
> > > > > > >> > > > optimum usage of the cluster nodes. Probably no static
> > > > > > >> configuration of
> > > > > > >> > > > Master-Slave in the load balancer nor in the hazelcast.
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > > But this would not allow to have the hotdeployment via
> > > > filesystem
> > > > > > >> still
> > > > > > >> > > enabled, right?
> > > > > > >> > >
> > > > > > >> > > Best,
> > > > > > >> > >   Tammo
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > Tammo van Lessen - http://www.taval.de
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Tammo van Lessen - http://www.taval.de
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi,

I will explain my approach as much as possible. The oldest node in the
hazelcast cluster is elected as the master node. In the failure of the
master node, next oldest node will be elected as the master node. This
master-slave configuration is just for deployment. When the hazelcast
cluster elected the master node, that node becomes a master node for
deploying process. So it will do the deploying artifacts. If you want to
get the idea of electing master node please refer the code which I have
located in the github. (
https://github.com/Subasinghe/ode/tree/ode_clustering)

I identified separated actions which should be followed by the master and
salve nodes.
Actions which are followed by master node only
1) create deployment unit
2) set the version nu to deployment unit
3) compile deployment unit
4) scan deployment unit
5) retire previous versions
Master node and slave nodes should create _processes which stores
ProcessConfImpl
Only master node will write the version nu to database, create .deployed
file

So there are some actions which should be followed only by master node
while other actions should be followed by all the nodes.The idea of having
a master node is deploying artifacts and avoid others from writing the
version nu to database.
Whether a node is active or passive, all nodes should do the deployment.Master
and slaves will follow necessary actions as in above.





On 27 May 2015 at 15:49, Sathwik B P <sa...@gmail.com> wrote:

> Nandika,
>
> I very well understand what you have put across, but it's secondary to me
> now.
>
> Sudharma,
> My primary concern is to understand at a high level the deployment
> architecture and how would master-slave configuration fit in. Are there any
> restrictions imposed by the in-progress design?
>
> Firstly, how would ODE process deployment work under these cluster
> configurations?
>
> Sample Cluster configurations: A load balancer is frontending the servers.
> 1) Cluster consisting of 2 nodes all Active-Active.
> 2) Cluster consisting of 2 nodes Active-Passive.
> 3) Cluster with 2+ nodes with additional nodes either in Active or Passive.
>
> regards,
> sathwik
>
>
>
>
>
>
>
> On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <ja...@gmail.com>
> wrote:
>
> > Hi Sathwik,
> >
> > According to my understanding, in the clustering scenario, the master
> node
> > should perform all the deployment actions and the slave nodes also need
> to
> > perform some deployment actions. For example, the slave nodes also should
> > handle the process ACTIVATED event so that the process configuration is
> > added to the engine and necessary web services are created so that when
> the
> > load balancer send requests to any node in the cluster, it is ready to
> > accept those requests.
> >
> > Regards
> > Nandika
> >
> >
> >
> >
> > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> > wrote:
> >
> > > Sudharma,
> > >
> > > Where are you going to configure the master-slaves, is it in the web
> > > application level or at the load balancer?
> > >
> > > regards,
> > > sathwik
> > >
> > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > > suba.11@cse.mrt.ac.lk>
> > > wrote:
> > >
> > > > Hi Tammo,
> > > >
> > > > Can you suggest the best method from these to implement? As first I
> > > > suggested the master-slaves scenario I think it is easy to implement
> > than
> > > > distributed lock scenario. However if you can suggest one from these
> > two,
> > > > then I can think about it.
> > > >
> > > > Thank you
> > > >
> > > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com> wrote:
> > > >
> > > > > With respect to the hotdeployment,
> > > > >
> > > > > We can drop the deployment archive onto the deployment folder.
> Since
> > > the
> > > > > DeploymentPoller are acquiring the distributed lock for the
> > > > DeploymentUnit,
> > > > > only one of the nodes will get the lock and initiate the
> deployment.
> > > > > DeploymentPollers on other nodes will fail in acquiring the lock
> and
> > > > hence
> > > > > will silently ignore it.
> > > > >
> > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> sathwik.bp@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Tammo,
> > > > > >
> > > > > > The distributed lock acquisition on the DeploymentUnit should be
> > > added
> > > > to
> > > > > > both DeploymentWebService and DeploymentPoller.
> > > > > >
> > > > > > When a deployment operation is initiated through the
> > > > > DeploymentWebService,
> > > > > > The load balancer routes it to any of the available nodes.
> > > > > >
> > > > > > On the routed node, the DeploymentWebService acquires the
> > Distributed
> > > > > > lock. On the remaining nodes the DeploymentPoller will try to
> > acquire
> > > > the
> > > > > > distributed lock and will not get it and hence will silently
> ignore
> > > it.
> > > > > >
> > > > > > Once the routed node completes the deployment, it will release
> the
> > > > lock.
> > > > > > This way we don't have to stall the DeploymentPoller in other
> > nodes.
> > > > > >
> > > > > > Does it answer the concerns?
> > > > > >
> > > > > >
> > > > > > Now, if we give the responsibility of identifying the master node
> > to
> > > > the
> > > > > > hazelcast, how do we plan to intimate the load balancer to change
> > > it's
> > > > > > configuration about the master node?
> > > > > > Assuming there are 3 nodes in the cluster,
> > > > > > node1 -master
> > > > > > node2 - slave
> > > > > > node3 - slave
> > > > > >
> > > > > > Node1 goes down, the LB will promote Node2 as master node, but
> > > > hazelcast
> > > > > > might promote Node3 as master node. They are out of sync.
> > > > > >
> > > > > > Is this argument valid?
> > > > > >
> > > > > > regards,
> > > > > > sathwik
> > > > > >
> > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > > > tvanlessen@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Sudharma,
> > > > > >>
> > > > > >> what do you expect from the "other nodes deployment"?
> Compilation
> > is
> > > > not
> > > > > >> needed since the CBP file is written to the (shared) FS.
> > > Registration
> > > > is
> > > > > >> also not needed, since it is done via the shared database. So
> the
> > > only
> > > > > >> thing that might be needed is to tell the engine that there is a
> > new
> > > > > >> deployment. I'd need to check that. If this is needed, I revert
> my
> > > > last
> > > > > >> statement, then it is perhaps better to just send an event over
> > > > > Hazelcast
> > > > > >> to all nodes that the deployment has changed.
> > > > > >>
> > > > > >> Best,
> > > > > >>   Tammo
> > > > > >>
> > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > > > > >> suba.11@cse.mrt.ac.lk
> > > > > >> > wrote:
> > > > > >>
> > > > > >> > Hi Tammo,
> > > > > >> >
> > > > > >> > The master node writes meta data. But runtime information must
> > be
> > > > > >> available
> > > > > >> > in all nodes.Since the folder is shared, all nodes will see
> the
> > > > > >> > availability of a new process. My idea is for master node to
> > write
> > > > the
> > > > > >> meta
> > > > > >> > data and other nodes to just read the meta data and load
> > > process.So
> > > > we
> > > > > >> need
> > > > > >> > a small delay between master node deployment and other nodes
> > > > > deployment.
> > > > > >> >
> > > > > >> > Is there anyway to set the delay between master node and
> slaves
> > > > until
> > > > > >> > master node finish the deployment?
> > > > > >> >
> > > > > >> > Thank you
> > > > > >> > Sudharma
> > > > > >> >
> > > > > >> >
> > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
> tvanlessen@gmail.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Hi Sathwik,
> > > > > >> > >
> > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > > > sathwik.bp@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > > Sudharma/Tammo,
> > > > > >> > > >
> > > > > >> > > > 1) How do we plan to decide which is the master node in
> the
> > > > > cluster?
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > > I think the easiest approach is to always elect the oldest
> > node
> > > in
> > > > > the
> > > > > >> > > cluster to be the master. AFAIK Hazelcast can easily asked
> for
> > > > this
> > > > > >> > > information.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > > 2) Don't we need to stall the Deployment Pollers in the
> > slave
> > > > > nodes?
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > Absolutely.
> > > > > >> > >
> > > > > >> > > Suggestion:
> > > > > >> > > > I am not sure whether do we need Master-SLaves. Why not
> give
> > > > every
> > > > > >> node
> > > > > >> > > in
> > > > > >> > > > the cluster the same status (Active-Active).
> > > > > >> > > >
> > > > > >> > > > When a new deployment is made, the load balancer can push
> it
> > > to
> > > > > any
> > > > > >> of
> > > > > >> > > the
> > > > > >> > > > available nodes. That node will probably acquire a
> > distributed
> > > > > lock
> > > > > >> on
> > > > > >> > > the
> > > > > >> > > > deployment unit and acts as master for that deployment.
> This
> > > > > ensures
> > > > > >> > > > optimum usage of the cluster nodes. Probably no static
> > > > > >> configuration of
> > > > > >> > > > Master-Slave in the load balancer nor in the hazelcast.
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > > But this would not allow to have the hotdeployment via
> > > filesystem
> > > > > >> still
> > > > > >> > > enabled, right?
> > > > > >> > >
> > > > > >> > > Best,
> > > > > >> > >   Tammo
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > Tammo van Lessen - http://www.taval.de
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Tammo van Lessen - http://www.taval.de
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Nandika,

I very well understand what you have put across, but it's secondary to me
now.

Sudharma,
My primary concern is to understand at a high level the deployment
architecture and how would master-slave configuration fit in. Are there any
restrictions imposed by the in-progress design?

Firstly, how would ODE process deployment work under these cluster
configurations?

Sample Cluster configurations: A load balancer is frontending the servers.
1) Cluster consisting of 2 nodes all Active-Active.
2) Cluster consisting of 2 nodes Active-Passive.
3) Cluster with 2+ nodes with additional nodes either in Active or Passive.

regards,
sathwik







On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <ja...@gmail.com>
wrote:

> Hi Sathwik,
>
> According to my understanding, in the clustering scenario, the master node
> should perform all the deployment actions and the slave nodes also need to
> perform some deployment actions. For example, the slave nodes also should
> handle the process ACTIVATED event so that the process configuration is
> added to the engine and necessary web services are created so that when the
> load balancer send requests to any node in the cluster, it is ready to
> accept those requests.
>
> Regards
> Nandika
>
>
>
>
> On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> wrote:
>
> > Sudharma,
> >
> > Where are you going to configure the master-slaves, is it in the web
> > application level or at the load balancer?
> >
> > regards,
> > sathwik
> >
> > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > suba.11@cse.mrt.ac.lk>
> > wrote:
> >
> > > Hi Tammo,
> > >
> > > Can you suggest the best method from these to implement? As first I
> > > suggested the master-slaves scenario I think it is easy to implement
> than
> > > distributed lock scenario. However if you can suggest one from these
> two,
> > > then I can think about it.
> > >
> > > Thank you
> > >
> > > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com> wrote:
> > >
> > > > With respect to the hotdeployment,
> > > >
> > > > We can drop the deployment archive onto the deployment folder. Since
> > the
> > > > DeploymentPoller are acquiring the distributed lock for the
> > > DeploymentUnit,
> > > > only one of the nodes will get the lock and initiate the deployment.
> > > > DeploymentPollers on other nodes will fail in acquiring the lock and
> > > hence
> > > > will silently ignore it.
> > > >
> > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Tammo,
> > > > >
> > > > > The distributed lock acquisition on the DeploymentUnit should be
> > added
> > > to
> > > > > both DeploymentWebService and DeploymentPoller.
> > > > >
> > > > > When a deployment operation is initiated through the
> > > > DeploymentWebService,
> > > > > The load balancer routes it to any of the available nodes.
> > > > >
> > > > > On the routed node, the DeploymentWebService acquires the
> Distributed
> > > > > lock. On the remaining nodes the DeploymentPoller will try to
> acquire
> > > the
> > > > > distributed lock and will not get it and hence will silently ignore
> > it.
> > > > >
> > > > > Once the routed node completes the deployment, it will release the
> > > lock.
> > > > > This way we don't have to stall the DeploymentPoller in other
> nodes.
> > > > >
> > > > > Does it answer the concerns?
> > > > >
> > > > >
> > > > > Now, if we give the responsibility of identifying the master node
> to
> > > the
> > > > > hazelcast, how do we plan to intimate the load balancer to change
> > it's
> > > > > configuration about the master node?
> > > > > Assuming there are 3 nodes in the cluster,
> > > > > node1 -master
> > > > > node2 - slave
> > > > > node3 - slave
> > > > >
> > > > > Node1 goes down, the LB will promote Node2 as master node, but
> > > hazelcast
> > > > > might promote Node3 as master node. They are out of sync.
> > > > >
> > > > > Is this argument valid?
> > > > >
> > > > > regards,
> > > > > sathwik
> > > > >
> > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > > tvanlessen@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Sudharma,
> > > > >>
> > > > >> what do you expect from the "other nodes deployment"? Compilation
> is
> > > not
> > > > >> needed since the CBP file is written to the (shared) FS.
> > Registration
> > > is
> > > > >> also not needed, since it is done via the shared database. So the
> > only
> > > > >> thing that might be needed is to tell the engine that there is a
> new
> > > > >> deployment. I'd need to check that. If this is needed, I revert my
> > > last
> > > > >> statement, then it is perhaps better to just send an event over
> > > > Hazelcast
> > > > >> to all nodes that the deployment has changed.
> > > > >>
> > > > >> Best,
> > > > >>   Tammo
> > > > >>
> > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > > > >> suba.11@cse.mrt.ac.lk
> > > > >> > wrote:
> > > > >>
> > > > >> > Hi Tammo,
> > > > >> >
> > > > >> > The master node writes meta data. But runtime information must
> be
> > > > >> available
> > > > >> > in all nodes.Since the folder is shared, all nodes will see the
> > > > >> > availability of a new process. My idea is for master node to
> write
> > > the
> > > > >> meta
> > > > >> > data and other nodes to just read the meta data and load
> > process.So
> > > we
> > > > >> need
> > > > >> > a small delay between master node deployment and other nodes
> > > > deployment.
> > > > >> >
> > > > >> > Is there anyway to set the delay between master node and slaves
> > > until
> > > > >> > master node finish the deployment?
> > > > >> >
> > > > >> > Thank you
> > > > >> > Sudharma
> > > > >> >
> > > > >> >
> > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <tvanlessen@gmail.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Hi Sathwik,
> > > > >> > >
> > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > > sathwik.bp@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > Sudharma/Tammo,
> > > > >> > > >
> > > > >> > > > 1) How do we plan to decide which is the master node in the
> > > > cluster?
> > > > >> > > >
> > > > >> > >
> > > > >> > > I think the easiest approach is to always elect the oldest
> node
> > in
> > > > the
> > > > >> > > cluster to be the master. AFAIK Hazelcast can easily asked for
> > > this
> > > > >> > > information.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > > 2) Don't we need to stall the Deployment Pollers in the
> slave
> > > > nodes?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > Absolutely.
> > > > >> > >
> > > > >> > > Suggestion:
> > > > >> > > > I am not sure whether do we need Master-SLaves. Why not give
> > > every
> > > > >> node
> > > > >> > > in
> > > > >> > > > the cluster the same status (Active-Active).
> > > > >> > > >
> > > > >> > > > When a new deployment is made, the load balancer can push it
> > to
> > > > any
> > > > >> of
> > > > >> > > the
> > > > >> > > > available nodes. That node will probably acquire a
> distributed
> > > > lock
> > > > >> on
> > > > >> > > the
> > > > >> > > > deployment unit and acts as master for that deployment. This
> > > > ensures
> > > > >> > > > optimum usage of the cluster nodes. Probably no static
> > > > >> configuration of
> > > > >> > > > Master-Slave in the load balancer nor in the hazelcast.
> > > > >> > > >
> > > > >> > >
> > > > >> > > But this would not allow to have the hotdeployment via
> > filesystem
> > > > >> still
> > > > >> > > enabled, right?
> > > > >> > >
> > > > >> > > Best,
> > > > >> > >   Tammo
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Tammo van Lessen - http://www.taval.de
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Tammo van Lessen - http://www.taval.de
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by Nandika Jayawardana <ja...@gmail.com>.

Hi Sathwik,

According to my understanding, in the clustering scenario, the master node
should perform all the deployment actions and the slave nodes also need to
perform some deployment actions. For example, the slave nodes also should
handle the process ACTIVATED event so that the process configuration is
added to the engine and necessary web services are created so that when the
load balancer send requests to any node in the cluster, it is ready to
accept those requests.

Regards
Nandika




On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com> wrote:

> Sudharma,
>
> Where are you going to configure the master-slaves, is it in the web
> application level or at the load balancer?
>
> regards,
> sathwik
>
> On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk>
> wrote:
>
> > Hi Tammo,
> >
> > Can you suggest the best method from these to implement? As first I
> > suggested the master-slaves scenario I think it is easy to implement than
> > distributed lock scenario. However if you can suggest one from these two,
> > then I can think about it.
> >
> > Thank you
> >
> > On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com> wrote:
> >
> > > With respect to the hotdeployment,
> > >
> > > We can drop the deployment archive onto the deployment folder. Since
> the
> > > DeploymentPoller are acquiring the distributed lock for the
> > DeploymentUnit,
> > > only one of the nodes will get the lock and initiate the deployment.
> > > DeploymentPollers on other nodes will fail in acquiring the lock and
> > hence
> > > will silently ignore it.
> > >
> > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Tammo,
> > > >
> > > > The distributed lock acquisition on the DeploymentUnit should be
> added
> > to
> > > > both DeploymentWebService and DeploymentPoller.
> > > >
> > > > When a deployment operation is initiated through the
> > > DeploymentWebService,
> > > > The load balancer routes it to any of the available nodes.
> > > >
> > > > On the routed node, the DeploymentWebService acquires the Distributed
> > > > lock. On the remaining nodes the DeploymentPoller will try to acquire
> > the
> > > > distributed lock and will not get it and hence will silently ignore
> it.
> > > >
> > > > Once the routed node completes the deployment, it will release the
> > lock.
> > > > This way we don't have to stall the DeploymentPoller in other nodes.
> > > >
> > > > Does it answer the concerns?
> > > >
> > > >
> > > > Now, if we give the responsibility of identifying the master node to
> > the
> > > > hazelcast, how do we plan to intimate the load balancer to change
> it's
> > > > configuration about the master node?
> > > > Assuming there are 3 nodes in the cluster,
> > > > node1 -master
> > > > node2 - slave
> > > > node3 - slave
> > > >
> > > > Node1 goes down, the LB will promote Node2 as master node, but
> > hazelcast
> > > > might promote Node3 as master node. They are out of sync.
> > > >
> > > > Is this argument valid?
> > > >
> > > > regards,
> > > > sathwik
> > > >
> > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> > tvanlessen@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Sudharma,
> > > >>
> > > >> what do you expect from the "other nodes deployment"? Compilation is
> > not
> > > >> needed since the CBP file is written to the (shared) FS.
> Registration
> > is
> > > >> also not needed, since it is done via the shared database. So the
> only
> > > >> thing that might be needed is to tell the engine that there is a new
> > > >> deployment. I'd need to check that. If this is needed, I revert my
> > last
> > > >> statement, then it is perhaps better to just send an event over
> > > Hazelcast
> > > >> to all nodes that the deployment has changed.
> > > >>
> > > >> Best,
> > > >>   Tammo
> > > >>
> > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > > >> suba.11@cse.mrt.ac.lk
> > > >> > wrote:
> > > >>
> > > >> > Hi Tammo,
> > > >> >
> > > >> > The master node writes meta data. But runtime information must be
> > > >> available
> > > >> > in all nodes.Since the folder is shared, all nodes will see the
> > > >> > availability of a new process. My idea is for master node to write
> > the
> > > >> meta
> > > >> > data and other nodes to just read the meta data and load
> process.So
> > we
> > > >> need
> > > >> > a small delay between master node deployment and other nodes
> > > deployment.
> > > >> >
> > > >> > Is there anyway to set the delay between master node and slaves
> > until
> > > >> > master node finish the deployment?
> > > >> >
> > > >> > Thank you
> > > >> > Sudharma
> > > >> >
> > > >> >
> > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Hi Sathwik,
> > > >> > >
> > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> > sathwik.bp@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > Sudharma/Tammo,
> > > >> > > >
> > > >> > > > 1) How do we plan to decide which is the master node in the
> > > cluster?
> > > >> > > >
> > > >> > >
> > > >> > > I think the easiest approach is to always elect the oldest node
> in
> > > the
> > > >> > > cluster to be the master. AFAIK Hazelcast can easily asked for
> > this
> > > >> > > information.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > > 2) Don't we need to stall the Deployment Pollers in the slave
> > > nodes?
> > > >> > > >
> > > >> > > >
> > > >> > > Absolutely.
> > > >> > >
> > > >> > > Suggestion:
> > > >> > > > I am not sure whether do we need Master-SLaves. Why not give
> > every
> > > >> node
> > > >> > > in
> > > >> > > > the cluster the same status (Active-Active).
> > > >> > > >
> > > >> > > > When a new deployment is made, the load balancer can push it
> to
> > > any
> > > >> of
> > > >> > > the
> > > >> > > > available nodes. That node will probably acquire a distributed
> > > lock
> > > >> on
> > > >> > > the
> > > >> > > > deployment unit and acts as master for that deployment. This
> > > ensures
> > > >> > > > optimum usage of the cluster nodes. Probably no static
> > > >> configuration of
> > > >> > > > Master-Slave in the load balancer nor in the hazelcast.
> > > >> > > >
> > > >> > >
> > > >> > > But this would not allow to have the hotdeployment via
> filesystem
> > > >> still
> > > >> > > enabled, right?
> > > >> > >
> > > >> > > Best,
> > > >> > >   Tammo
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Tammo van Lessen - http://www.taval.de
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Tammo van Lessen - http://www.taval.de
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Sudharma,

Where are you going to configure the master-slaves, is it in the web
application level or at the load balancer?

regards,
sathwik

On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <su...@cse.mrt.ac.lk>
wrote:

> Hi Tammo,
>
> Can you suggest the best method from these to implement? As first I
> suggested the master-slaves scenario I think it is easy to implement than
> distributed lock scenario. However if you can suggest one from these two,
> then I can think about it.
>
> Thank you
>
> On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com> wrote:
>
> > With respect to the hotdeployment,
> >
> > We can drop the deployment archive onto the deployment folder. Since the
> > DeploymentPoller are acquiring the distributed lock for the
> DeploymentUnit,
> > only one of the nodes will get the lock and initiate the deployment.
> > DeploymentPollers on other nodes will fail in acquiring the lock and
> hence
> > will silently ignore it.
> >
> > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> > wrote:
> >
> > > Hi Tammo,
> > >
> > > The distributed lock acquisition on the DeploymentUnit should be added
> to
> > > both DeploymentWebService and DeploymentPoller.
> > >
> > > When a deployment operation is initiated through the
> > DeploymentWebService,
> > > The load balancer routes it to any of the available nodes.
> > >
> > > On the routed node, the DeploymentWebService acquires the Distributed
> > > lock. On the remaining nodes the DeploymentPoller will try to acquire
> the
> > > distributed lock and will not get it and hence will silently ignore it.
> > >
> > > Once the routed node completes the deployment, it will release the
> lock.
> > > This way we don't have to stall the DeploymentPoller in other nodes.
> > >
> > > Does it answer the concerns?
> > >
> > >
> > > Now, if we give the responsibility of identifying the master node to
> the
> > > hazelcast, how do we plan to intimate the load balancer to change it's
> > > configuration about the master node?
> > > Assuming there are 3 nodes in the cluster,
> > > node1 -master
> > > node2 - slave
> > > node3 - slave
> > >
> > > Node1 goes down, the LB will promote Node2 as master node, but
> hazelcast
> > > might promote Node3 as master node. They are out of sync.
> > >
> > > Is this argument valid?
> > >
> > > regards,
> > > sathwik
> > >
> > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <
> tvanlessen@gmail.com>
> > > wrote:
> > >
> > >> Hi Sudharma,
> > >>
> > >> what do you expect from the "other nodes deployment"? Compilation is
> not
> > >> needed since the CBP file is written to the (shared) FS. Registration
> is
> > >> also not needed, since it is done via the shared database. So the only
> > >> thing that might be needed is to tell the engine that there is a new
> > >> deployment. I'd need to check that. If this is needed, I revert my
> last
> > >> statement, then it is perhaps better to just send an event over
> > Hazelcast
> > >> to all nodes that the deployment has changed.
> > >>
> > >> Best,
> > >>   Tammo
> > >>
> > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> > >> suba.11@cse.mrt.ac.lk
> > >> > wrote:
> > >>
> > >> > Hi Tammo,
> > >> >
> > >> > The master node writes meta data. But runtime information must be
> > >> available
> > >> > in all nodes.Since the folder is shared, all nodes will see the
> > >> > availability of a new process. My idea is for master node to write
> the
> > >> meta
> > >> > data and other nodes to just read the meta data and load process.So
> we
> > >> need
> > >> > a small delay between master node deployment and other nodes
> > deployment.
> > >> >
> > >> > Is there anyway to set the delay between master node and slaves
> until
> > >> > master node finish the deployment?
> > >> >
> > >> > Thank you
> > >> > Sudharma
> > >> >
> > >> >
> > >> > On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hi Sathwik,
> > >> > >
> > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <
> sathwik.bp@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > Sudharma/Tammo,
> > >> > > >
> > >> > > > 1) How do we plan to decide which is the master node in the
> > cluster?
> > >> > > >
> > >> > >
> > >> > > I think the easiest approach is to always elect the oldest node in
> > the
> > >> > > cluster to be the master. AFAIK Hazelcast can easily asked for
> this
> > >> > > information.
> > >> > >
> > >> > >
> > >> > >
> > >> > > > 2) Don't we need to stall the Deployment Pollers in the slave
> > nodes?
> > >> > > >
> > >> > > >
> > >> > > Absolutely.
> > >> > >
> > >> > > Suggestion:
> > >> > > > I am not sure whether do we need Master-SLaves. Why not give
> every
> > >> node
> > >> > > in
> > >> > > > the cluster the same status (Active-Active).
> > >> > > >
> > >> > > > When a new deployment is made, the load balancer can push it to
> > any
> > >> of
> > >> > > the
> > >> > > > available nodes. That node will probably acquire a distributed
> > lock
> > >> on
> > >> > > the
> > >> > > > deployment unit and acts as master for that deployment. This
> > ensures
> > >> > > > optimum usage of the cluster nodes. Probably no static
> > >> configuration of
> > >> > > > Master-Slave in the load balancer nor in the hazelcast.
> > >> > > >
> > >> > >
> > >> > > But this would not allow to have the hotdeployment via filesystem
> > >> still
> > >> > > enabled, right?
> > >> > >
> > >> > > Best,
> > >> > >   Tammo
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Tammo van Lessen - http://www.taval.de
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Tammo van Lessen - http://www.taval.de
> > >>
> > >
> > >
> >
>

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Tammo,

Can you suggest the best method from these to implement? As first I
suggested the master-slaves scenario I think it is easy to implement than
distributed lock scenario. However if you can suggest one from these two,
then I can think about it.

Thank you

On 21 May 2015 at 12:40, Sathwik B P <sa...@gmail.com> wrote:

> With respect to the hotdeployment,
>
> We can drop the deployment archive onto the deployment folder. Since the
> DeploymentPoller are acquiring the distributed lock for the DeploymentUnit,
> only one of the nodes will get the lock and initiate the deployment.
> DeploymentPollers on other nodes will fail in acquiring the lock and hence
> will silently ignore it.
>
> On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com>
> wrote:
>
> > Hi Tammo,
> >
> > The distributed lock acquisition on the DeploymentUnit should be added to
> > both DeploymentWebService and DeploymentPoller.
> >
> > When a deployment operation is initiated through the
> DeploymentWebService,
> > The load balancer routes it to any of the available nodes.
> >
> > On the routed node, the DeploymentWebService acquires the Distributed
> > lock. On the remaining nodes the DeploymentPoller will try to acquire the
> > distributed lock and will not get it and hence will silently ignore it.
> >
> > Once the routed node completes the deployment, it will release the lock.
> > This way we don't have to stall the DeploymentPoller in other nodes.
> >
> > Does it answer the concerns?
> >
> >
> > Now, if we give the responsibility of identifying the master node to the
> > hazelcast, how do we plan to intimate the load balancer to change it's
> > configuration about the master node?
> > Assuming there are 3 nodes in the cluster,
> > node1 -master
> > node2 - slave
> > node3 - slave
> >
> > Node1 goes down, the LB will promote Node2 as master node, but hazelcast
> > might promote Node3 as master node. They are out of sync.
> >
> > Is this argument valid?
> >
> > regards,
> > sathwik
> >
> > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <tv...@gmail.com>
> > wrote:
> >
> >> Hi Sudharma,
> >>
> >> what do you expect from the "other nodes deployment"? Compilation is not
> >> needed since the CBP file is written to the (shared) FS. Registration is
> >> also not needed, since it is done via the shared database. So the only
> >> thing that might be needed is to tell the engine that there is a new
> >> deployment. I'd need to check that. If this is needed, I revert my last
> >> statement, then it is perhaps better to just send an event over
> Hazelcast
> >> to all nodes that the deployment has changed.
> >>
> >> Best,
> >>   Tammo
> >>
> >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> >> suba.11@cse.mrt.ac.lk
> >> > wrote:
> >>
> >> > Hi Tammo,
> >> >
> >> > The master node writes meta data. But runtime information must be
> >> available
> >> > in all nodes.Since the folder is shared, all nodes will see the
> >> > availability of a new process. My idea is for master node to write the
> >> meta
> >> > data and other nodes to just read the meta data and load process.So we
> >> need
> >> > a small delay between master node deployment and other nodes
> deployment.
> >> >
> >> > Is there anyway to set the delay between master node and slaves until
> >> > master node finish the deployment?
> >> >
> >> > Thank you
> >> > Sudharma
> >> >
> >> >
> >> > On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com>
> wrote:
> >> >
> >> > > Hi Sathwik,
> >> > >
> >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <sa...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Sudharma/Tammo,
> >> > > >
> >> > > > 1) How do we plan to decide which is the master node in the
> cluster?
> >> > > >
> >> > >
> >> > > I think the easiest approach is to always elect the oldest node in
> the
> >> > > cluster to be the master. AFAIK Hazelcast can easily asked for this
> >> > > information.
> >> > >
> >> > >
> >> > >
> >> > > > 2) Don't we need to stall the Deployment Pollers in the slave
> nodes?
> >> > > >
> >> > > >
> >> > > Absolutely.
> >> > >
> >> > > Suggestion:
> >> > > > I am not sure whether do we need Master-SLaves. Why not give every
> >> node
> >> > > in
> >> > > > the cluster the same status (Active-Active).
> >> > > >
> >> > > > When a new deployment is made, the load balancer can push it to
> any
> >> of
> >> > > the
> >> > > > available nodes. That node will probably acquire a distributed
> lock
> >> on
> >> > > the
> >> > > > deployment unit and acts as master for that deployment. This
> ensures
> >> > > > optimum usage of the cluster nodes. Probably no static
> >> configuration of
> >> > > > Master-Slave in the load balancer nor in the hazelcast.
> >> > > >
> >> > >
> >> > > But this would not allow to have the hotdeployment via filesystem
> >> still
> >> > > enabled, right?
> >> > >
> >> > > Best,
> >> > >   Tammo
> >> > >
> >> > >
> >> > > --
> >> > > Tammo van Lessen - http://www.taval.de
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Tammo van Lessen - http://www.taval.de
> >>
> >
> >
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

With respect to the hotdeployment,

We can drop the deployment archive onto the deployment folder. Since the
DeploymentPoller are acquiring the distributed lock for the DeploymentUnit,
only one of the nodes will get the lock and initiate the deployment.
DeploymentPollers on other nodes will fail in acquiring the lock and hence
will silently ignore it.

On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <sa...@gmail.com> wrote:

> Hi Tammo,
>
> The distributed lock acquisition on the DeploymentUnit should be added to
> both DeploymentWebService and DeploymentPoller.
>
> When a deployment operation is initiated through the DeploymentWebService,
> The load balancer routes it to any of the available nodes.
>
> On the routed node, the DeploymentWebService acquires the Distributed
> lock. On the remaining nodes the DeploymentPoller will try to acquire the
> distributed lock and will not get it and hence will silently ignore it.
>
> Once the routed node completes the deployment, it will release the lock.
> This way we don't have to stall the DeploymentPoller in other nodes.
>
> Does it answer the concerns?
>
>
> Now, if we give the responsibility of identifying the master node to the
> hazelcast, how do we plan to intimate the load balancer to change it's
> configuration about the master node?
> Assuming there are 3 nodes in the cluster,
> node1 -master
> node2 - slave
> node3 - slave
>
> Node1 goes down, the LB will promote Node2 as master node, but hazelcast
> might promote Node3 as master node. They are out of sync.
>
> Is this argument valid?
>
> regards,
> sathwik
>
> On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <tv...@gmail.com>
> wrote:
>
>> Hi Sudharma,
>>
>> what do you expect from the "other nodes deployment"? Compilation is not
>> needed since the CBP file is written to the (shared) FS. Registration is
>> also not needed, since it is done via the shared database. So the only
>> thing that might be needed is to tell the engine that there is a new
>> deployment. I'd need to check that. If this is needed, I revert my last
>> statement, then it is perhaps better to just send an event over Hazelcast
>> to all nodes that the deployment has changed.
>>
>> Best,
>>   Tammo
>>
>> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
>> suba.11@cse.mrt.ac.lk
>> > wrote:
>>
>> > Hi Tammo,
>> >
>> > The master node writes meta data. But runtime information must be
>> available
>> > in all nodes.Since the folder is shared, all nodes will see the
>> > availability of a new process. My idea is for master node to write the
>> meta
>> > data and other nodes to just read the meta data and load process.So we
>> need
>> > a small delay between master node deployment and other nodes deployment.
>> >
>> > Is there anyway to set the delay between master node and slaves until
>> > master node finish the deployment?
>> >
>> > Thank you
>> > Sudharma
>> >
>> >
>> > On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com> wrote:
>> >
>> > > Hi Sathwik,
>> > >
>> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <sa...@gmail.com>
>> > wrote:
>> > >
>> > > > Sudharma/Tammo,
>> > > >
>> > > > 1) How do we plan to decide which is the master node in the cluster?
>> > > >
>> > >
>> > > I think the easiest approach is to always elect the oldest node in the
>> > > cluster to be the master. AFAIK Hazelcast can easily asked for this
>> > > information.
>> > >
>> > >
>> > >
>> > > > 2) Don't we need to stall the Deployment Pollers in the slave nodes?
>> > > >
>> > > >
>> > > Absolutely.
>> > >
>> > > Suggestion:
>> > > > I am not sure whether do we need Master-SLaves. Why not give every
>> node
>> > > in
>> > > > the cluster the same status (Active-Active).
>> > > >
>> > > > When a new deployment is made, the load balancer can push it to any
>> of
>> > > the
>> > > > available nodes. That node will probably acquire a distributed lock
>> on
>> > > the
>> > > > deployment unit and acts as master for that deployment. This ensures
>> > > > optimum usage of the cluster nodes. Probably no static
>> configuration of
>> > > > Master-Slave in the load balancer nor in the hazelcast.
>> > > >
>> > >
>> > > But this would not allow to have the hotdeployment via filesystem
>> still
>> > > enabled, right?
>> > >
>> > > Best,
>> > >   Tammo
>> > >
>> > >
>> > > --
>> > > Tammo van Lessen - http://www.taval.de
>> > >
>> >
>>
>>
>>
>> --
>> Tammo van Lessen - http://www.taval.de
>>
>
>

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Hi Tammo,

The distributed lock acquisition on the DeploymentUnit should be added to
both DeploymentWebService and DeploymentPoller.

When a deployment operation is initiated through the DeploymentWebService,
The load balancer routes it to any of the available nodes.

On the routed node, the DeploymentWebService acquires the Distributed lock.
On the remaining nodes the DeploymentPoller will try to acquire the
distributed lock and will not get it and hence will silently ignore it.

Once the routed node completes the deployment, it will release the lock.
This way we don't have to stall the DeploymentPoller in other nodes.

Does it answer the concerns?


Now, if we give the responsibility of identifying the master node to the
hazelcast, how do we plan to intimate the load balancer to change it's
configuration about the master node?
Assuming there are 3 nodes in the cluster,
node1 -master
node2 - slave
node3 - slave

Node1 goes down, the LB will promote Node2 as master node, but hazelcast
might promote Node3 as master node. They are out of sync.

Is this argument valid?

regards,
sathwik

On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen <tv...@gmail.com>
wrote:

> Hi Sudharma,
>
> what do you expect from the "other nodes deployment"? Compilation is not
> needed since the CBP file is written to the (shared) FS. Registration is
> also not needed, since it is done via the shared database. So the only
> thing that might be needed is to tell the engine that there is a new
> deployment. I'd need to check that. If this is needed, I revert my last
> statement, then it is perhaps better to just send an event over Hazelcast
> to all nodes that the deployment has changed.
>
> Best,
>   Tammo
>
> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk
> > wrote:
>
> > Hi Tammo,
> >
> > The master node writes meta data. But runtime information must be
> available
> > in all nodes.Since the folder is shared, all nodes will see the
> > availability of a new process. My idea is for master node to write the
> meta
> > data and other nodes to just read the meta data and load process.So we
> need
> > a small delay between master node deployment and other nodes deployment.
> >
> > Is there anyway to set the delay between master node and slaves until
> > master node finish the deployment?
> >
> > Thank you
> > Sudharma
> >
> >
> > On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com> wrote:
> >
> > > Hi Sathwik,
> > >
> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <sa...@gmail.com>
> > wrote:
> > >
> > > > Sudharma/Tammo,
> > > >
> > > > 1) How do we plan to decide which is the master node in the cluster?
> > > >
> > >
> > > I think the easiest approach is to always elect the oldest node in the
> > > cluster to be the master. AFAIK Hazelcast can easily asked for this
> > > information.
> > >
> > >
> > >
> > > > 2) Don't we need to stall the Deployment Pollers in the slave nodes?
> > > >
> > > >
> > > Absolutely.
> > >
> > > Suggestion:
> > > > I am not sure whether do we need Master-SLaves. Why not give every
> node
> > > in
> > > > the cluster the same status (Active-Active).
> > > >
> > > > When a new deployment is made, the load balancer can push it to any
> of
> > > the
> > > > available nodes. That node will probably acquire a distributed lock
> on
> > > the
> > > > deployment unit and acts as master for that deployment. This ensures
> > > > optimum usage of the cluster nodes. Probably no static configuration
> of
> > > > Master-Slave in the load balancer nor in the hazelcast.
> > > >
> > >
> > > But this would not allow to have the hotdeployment via filesystem still
> > > enabled, right?
> > >
> > > Best,
> > >   Tammo
> > >
> > >
> > > --
> > > Tammo van Lessen - http://www.taval.de
> > >
> >
>
>
>
> --
> Tammo van Lessen - http://www.taval.de
>

Re: Deploying process in the cluster

Posted by Tammo van Lessen <tv...@gmail.com>.

Hi Sudharma,

what do you expect from the "other nodes deployment"? Compilation is not
needed since the CBP file is written to the (shared) FS. Registration is
also not needed, since it is done via the shared database. So the only
thing that might be needed is to tell the engine that there is a new
deployment. I'd need to check that. If this is needed, I revert my last
statement, then it is perhaps better to just send an event over Hazelcast
to all nodes that the deployment has changed.

Best,
  Tammo

On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe <suba.11@cse.mrt.ac.lk
> wrote:

> Hi Tammo,
>
> The master node writes meta data. But runtime information must be available
> in all nodes.Since the folder is shared, all nodes will see the
> availability of a new process. My idea is for master node to write the meta
> data and other nodes to just read the meta data and load process.So we need
> a small delay between master node deployment and other nodes deployment.
>
> Is there anyway to set the delay between master node and slaves until
> master node finish the deployment?
>
> Thank you
> Sudharma
>
>
> On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com> wrote:
>
> > Hi Sathwik,
> >
> > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <sa...@gmail.com>
> wrote:
> >
> > > Sudharma/Tammo,
> > >
> > > 1) How do we plan to decide which is the master node in the cluster?
> > >
> >
> > I think the easiest approach is to always elect the oldest node in the
> > cluster to be the master. AFAIK Hazelcast can easily asked for this
> > information.
> >
> >
> >
> > > 2) Don't we need to stall the Deployment Pollers in the slave nodes?
> > >
> > >
> > Absolutely.
> >
> > Suggestion:
> > > I am not sure whether do we need Master-SLaves. Why not give every node
> > in
> > > the cluster the same status (Active-Active).
> > >
> > > When a new deployment is made, the load balancer can push it to any of
> > the
> > > available nodes. That node will probably acquire a distributed lock on
> > the
> > > deployment unit and acts as master for that deployment. This ensures
> > > optimum usage of the cluster nodes. Probably no static configuration of
> > > Master-Slave in the load balancer nor in the hazelcast.
> > >
> >
> > But this would not allow to have the hotdeployment via filesystem still
> > enabled, right?
> >
> > Best,
> >   Tammo
> >
> >
> > --
> > Tammo van Lessen - http://www.taval.de
> >
>



-- 
Tammo van Lessen - http://www.taval.de

Re: Deploying process in the cluster

Posted by sudharma subasinghe <su...@cse.mrt.ac.lk>.

Hi Tammo,

The master node writes meta data. But runtime information must be available
in all nodes.Since the folder is shared, all nodes will see the
availability of a new process. My idea is for master node to write the meta
data and other nodes to just read the meta data and load process.So we need
a small delay between master node deployment and other nodes deployment.

Is there anyway to set the delay between master node and slaves until
master node finish the deployment?

Thank you
Sudharma

On 20 May 2015 at 13:01, Tammo van Lessen <tv...@gmail.com> wrote:

> Hi Sathwik,
>
> On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <sa...@gmail.com> wrote:
>
> > Sudharma/Tammo,
> >
> > 1) How do we plan to decide which is the master node in the cluster?
> >
>
> I think the easiest approach is to always elect the oldest node in the
> cluster to be the master. AFAIK Hazelcast can easily asked for this
> information.
>
>
>
> > 2) Don't we need to stall the Deployment Pollers in the slave nodes?
> >
> >
> Absolutely.
>
> Suggestion:
> > I am not sure whether do we need Master-SLaves. Why not give every node
> in
> > the cluster the same status (Active-Active).
> >
> > When a new deployment is made, the load balancer can push it to any of
> the
> > available nodes. That node will probably acquire a distributed lock on
> the
> > deployment unit and acts as master for that deployment. This ensures
> > optimum usage of the cluster nodes. Probably no static configuration of
> > Master-Slave in the load balancer nor in the hazelcast.
> >
>
> But this would not allow to have the hotdeployment via filesystem still
> enabled, right?
>
> Best,
>   Tammo
>
>
> --
> Tammo van Lessen - http://www.taval.de
>

Re: Deploying process in the cluster

Posted by Tammo van Lessen <tv...@gmail.com>.

Hi Sathwik,

On Wed, May 20, 2015 at 6:40 AM, Sathwik B P <sa...@gmail.com> wrote:

> Sudharma/Tammo,
>
> 1) How do we plan to decide which is the master node in the cluster?
>

I think the easiest approach is to always elect the oldest node in the
cluster to be the master. AFAIK Hazelcast can easily asked for this
information.



> 2) Don't we need to stall the Deployment Pollers in the slave nodes?
>
>
Absolutely.

Suggestion:
> I am not sure whether do we need Master-SLaves. Why not give every node in
> the cluster the same status (Active-Active).
>
> When a new deployment is made, the load balancer can push it to any of the
> available nodes. That node will probably acquire a distributed lock on the
> deployment unit and acts as master for that deployment. This ensures
> optimum usage of the cluster nodes. Probably no static configuration of
> Master-Slave in the load balancer nor in the hazelcast.
>

But this would not allow to have the hotdeployment via filesystem still
enabled, right?

Best,
  Tammo


-- 
Tammo van Lessen - http://www.taval.de

Re: Deploying process in the cluster

Posted by Sathwik B P <sa...@gmail.com>.

Sudharma/Tammo,

1) How do we plan to decide which is the master node in the cluster?

2) Don't we need to stall the Deployment Pollers in the slave nodes?

Suggestion:
I am not sure whether do we need Master-SLaves. Why not give every node in
the cluster the same status (Active-Active).

When a new deployment is made, the load balancer can push it to any of the
available nodes. That node will probably acquire a distributed lock on the
deployment unit and acts as master for that deployment. This ensures
optimum usage of the cluster nodes. Probably no static configuration of
Master-Slave in the load balancer nor in the hazelcast.

On Tue, May 19, 2015 at 4:31 PM, Tammo van Lessen <tv...@gmail.com>
wrote:

> Hi Sudharma,
>
> Electing one master that is able to deploy is the right approach. Since the
> metadata of a deployment is written to the database and the database is
> shared among all nodes, sending notifications is not necessary. This is all
> under the assumption that all nodes share the same deployment folder, e.g.
> using SMB or NFS.
>
> HTH,
>   Tammo
>
> On Thu, May 14, 2015 at 11:59 AM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk
> > wrote:
>
> > Hi,
> >
> > In the cluster, every node shares the database and deployment directory.
> To
> > avoid race condition I am going to introduce master and slave
> > configuration. Here master node will deploy the process and write the
> > version number to the database. Until master node finishes its process
> all
> > the salve nodes will wait. After finishing the master node's task it will
> > send a custom message to slave nodes through the cluster. Then they can
> get
> > the version number from database and will read the deployed process to
> > deploy it in salve nodes. Please give me ideas on this.
> >
> > Thanks,
> > Sudharma
> >
>
>
>
> --
> Tammo van Lessen - http://www.taval.de
>

Re: Deploying process in the cluster

Posted by Tammo van Lessen <tv...@gmail.com>.

Hi Sudharma,

Electing one master that is able to deploy is the right approach. Since the
metadata of a deployment is written to the database and the database is
shared among all nodes, sending notifications is not necessary. This is all
under the assumption that all nodes share the same deployment folder, e.g.
using SMB or NFS.

HTH,
  Tammo

On Thu, May 14, 2015 at 11:59 AM, sudharma subasinghe <suba.11@cse.mrt.ac.lk
> wrote:

> Hi,
>
> In the cluster, every node shares the database and deployment directory. To
> avoid race condition I am going to introduce master and slave
> configuration. Here master node will deploy the process and write the
> version number to the database. Until master node finishes its process all
> the salve nodes will wait. After finishing the master node's task it will
> send a custom message to slave nodes through the cluster. Then they can get
> the version number from database and will read the deployed process to
> deploy it in salve nodes. Please give me ideas on this.
>
> Thanks,
> Sudharma
>

-- 
Tammo van Lessen - http://www.taval.de