You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by Afkham Azeez <af...@gmail.com> on 2007/05/30 15:01:08 UTC
[Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
I think there is no need for the contextAdded & contextRemoved methods in
the ContextManager interface. contextUpdated should suffice.
The reason is, when a contextUpdated message is received, if it does not
exist on a particular node, it can be created. Similary, the context cleanup
mechanism will take care of cleaning up the contexts on each node, hence
removeContext is also not needed. This also simplifies the clustering
implementation, since an updateContext message can be received before a
contextAdded message due to the order in which the threads run at a
particular time. Otherwise, the sender has to ensure that always a
createContext message is sent to the channel before an updateContext message
is sent.
Thoughts?
--
Thanks
Afkham Azeez
http://www.wso2.org
GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Afkham Azeez <af...@gmail.com>.
On 6/2/07, Rajith Attapattu <ra...@gmail.com> wrote:
>
> When I receive an update how would I know whether it was a create+update
> or a regular update ?
> The absence of a the context in the receiving node doesn't necessarily
> imply that it was create + update. It could be that this node was down when
> the real creation happened.
>
Exactly. This is why I've introduced a message to obtain the latest state
from a randomly selected member, when a node tries to join a cluster. So
again, I feel we do not need an addContext message.
--
Thanks
Afkham Azeez
http://www.wso2.org
GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Afkham Azeez <af...@gmail.com>.
Point taken.
Thanks
Azeez
On 6/5/07, Rajith Attapattu <ra...@gmail.com> wrote:
>
> Azeez, my main point was not about security and my bad for misleading u
> with the wrong example.
> The point I am trying to make is that the whole purpose of clustering is
> to keep a set of related nodes in a consistent state.
> My examples was more geared towards integrity and not security.
>
> If u only remove state from one node and not on the anothers then u are
> going to have an inconsistent cluster. Consider the same example node A,B
> and C. If u logout from A and before any cleanup is run, what if your
> cluster goes down??
> Now when the cluster comes back up and u are trying to recover. Now u have
> 2 nodes (B & C) which thinks the session is alive and one node (A) which
> thinks the session is dead. If u apply quorum the consensus in the cluster
> will be that the session is alive, which is totally wrong. Now u endup with
> an unreliable cluster. This may or may not do any harm , but this is beside
> the point. It's conceptually wrong!!!!
>
> In my previous example the point of confusion was "Now some other entity
> can point to Node B or C" which gave u the impression that it was a security
> breach. Lets say the client logs out, but for some reason retries some
> operation ( after the session destroyed) due to a bug in the client code
> and if the requests gets routed to Node B or C it will continue to work.
> Even though these are rare events these things should be prevented.
>
> And if there is a security breach then it should be plugged no matter how
> small it is. Yes we don't rely on this for security and yes we want
> WS-Security ..etc, but still it's a loop hole and it should be closed. It's
> not an excuse to have something conceptually wrong for the sake of
> performance. When u try to deploy this at a bank etc, this will fail a
> security audit. These people are paranoid and will require u to close every
> loop hole they see no matter how big or small they are. I am telling this
> through experience.
>
> Also what about this point?
> "Further any cleanup associated with the lifecycle of the session (ex
> resource release) will not be executed on Node B and C."
> Why are we holding on to resource when they are not needed?
>
> Sometimes the performance savings that u are suggesting will come at a
> greater cost than u think.
> We need to look at the bigger picture and we shouldn't compromise
> integrity/reliability etc for performance.
> whats the purpose of having clustering if your cluster is inconsistent?
>
> Regards,
>
> Rajith
>
> On 6/4/07, Afkham Azeez < afkham@gmail.com> wrote:
> >
> > On 6/2/07, Rajith Attapattu <ra...@gmail.com> wrote:Azeez>>
> >
> > > Similarly, there is a cleanup mechanism which runs whenever a new
> > > contexts gets added. Hence there is no need to send out a removeContext
> > > message.
> > >
> >
> > Rajith>>
> >
> > This is not good enough. We need to remove a context ASAP as soon as a
> > > client logs out. Consider this. Client logs into Node A and does some work.
> > > All information is replicated to Node B & C. The client logout from Node A.
> > > We don't replicate the remove command and no new contexts are added, hence
> > > no cleanup mechanism is executed.
> > > Now some other entity can point to Node B or C and continue the
> > > session as those contexts are still available in those nodes. Further any
> > > cleanup associated with the lifecycle of the session (ex resource release)
> > > will not be executed on Node B and C. These kind of situations will make the
> > > cluster unreliable.
> > >
> > > We can't compromise integrity for the sake of performance. We need to
> > > achieve both without compromising each other.
> > >
> > >
> > What about the situation where a session has been created on behalf of a
> > client and this session is replicated, and the session is still active? A
> > malicious entity can get hold of the session id and connect to any node. In
> > Axis2, there are 2 types of sessions; soapsession & transportsession. In the
> > case of the soapsession, the session is tracked using the service group
> > context id in the addressing header. If proper security is not used, anybody
> > can capture this id and masquerade as the legitimate client. Similary, in
> > transportsession, the SessionContext is associated with the session
> > management mechanism of the transport. Even in such a case, if proper
> > security at the transport level is not used, a masquerading attack can take
> > place. My point is, removing the session from the nodes is not the solution
> > for securing the system. We must use WS-Security to properly handle this
> > requirement.
> >
> >
> > --
> > Thanks
> > Afkham Azeez
> >
> > http://www.wso2.org
> > GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
> >
>
>
--
Thanks
Afkham Azeez
http://www.wso2.org
GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Rajith Attapattu <ra...@gmail.com>.
Azeez, my main point was not about security and my bad for misleading u with
the wrong example.
The point I am trying to make is that the whole purpose of clustering is to
keep a set of related nodes in a consistent state.
My examples was more geared towards integrity and not security.
If u only remove state from one node and not on the anothers then u are
going to have an inconsistent cluster. Consider the same example node A,B
and C. If u logout from A and before any cleanup is run, what if your
cluster goes down??
Now when the cluster comes back up and u are trying to recover. Now u have 2
nodes (B & C) which thinks the session is alive and one node (A) which
thinks the session is dead. If u apply quorum the consensus in the cluster
will be that the session is alive, which is totally wrong. Now u endup with
an unreliable cluster. This may or may not do any harm , but this is beside
the point. It's conceptually wrong!!!!
In my previous example the point of confusion was "Now some other entity can
point to Node B or C" which gave u the impression that it was a security
breach. Lets say the client logs out, but for some reason retries some
operation (after the session destroyed) due to a bug in the client code and
if the requests gets routed to Node B or C it will continue to work. Even
though these are rare events these things should be prevented.
And if there is a security breach then it should be plugged no matter how
small it is. Yes we don't rely on this for security and yes we want
WS-Security ..etc, but still it's a loop hole and it should be closed. It's
not an excuse to have something conceptually wrong for the sake of
performance. When u try to deploy this at a bank etc, this will fail a
security audit. These people are paranoid and will require u to close every
loop hole they see no matter how big or small they are. I am telling this
through experience.
Also what about this point?
"Further any cleanup associated with the lifecycle of the session (ex
resource release) will not be executed on Node B and C."
Why are we holding on to resource when they are not needed?
Sometimes the performance savings that u are suggesting will come at a
greater cost than u think.
We need to look at the bigger picture and we shouldn't compromise
integrity/reliability etc for performance.
whats the purpose of having clustering if your cluster is inconsistent?
Regards,
Rajith
On 6/4/07, Afkham Azeez <af...@gmail.com> wrote:
>
> On 6/2/07, Rajith Attapattu <ra...@gmail.com> wrote:Azeez>>
>
> > Similarly, there is a cleanup mechanism which runs whenever a new
> > contexts gets added. Hence there is no need to send out a removeContext
> > message.
> >
>
> Rajith>>
>
> This is not good enough. We need to remove a context ASAP as soon as a
> > client logs out. Consider this. Client logs into Node A and does some work.
> > All information is replicated to Node B & C. The client logout from Node A.
> > We don't replicate the remove command and no new contexts are added, hence
> > no cleanup mechanism is executed.
> > Now some other entity can point to Node B or C and continue the session
> > as those contexts are still available in those nodes. Further any cleanup
> > associated with the lifecycle of the session (ex resource release) will not
> > be executed on Node B and C. These kind of situations will make the cluster
> > unreliable.
> >
> > We can't compromise integrity for the sake of performance. We need to
> > achieve both without compromising each other.
> >
> >
> What about the situation where a session has been created on behalf of a
> client and this session is replicated, and the session is still active? A
> malicious entity can get hold of the session id and connect to any node. In
> Axis2, there are 2 types of sessions; soapsession & transportsession. In the
> case of the soapsession, the session is tracked using the service group
> context id in the addressing header. If proper security is not used, anybody
> can capture this id and masquerade as the legitimate client. Similary, in
> transportsession, the SessionContext is associated with the session
> management mechanism of the transport. Even in such a case, if proper
> security at the transport level is not used, a masquerading attack can take
> place. My point is, removing the session from the nodes is not the solution
> for securing the system. We must use WS-Security to properly handle this
> requirement.
>
>
> --
> Thanks
> Afkham Azeez
>
> http://www.wso2.org
> GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
>
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Afkham Azeez <af...@gmail.com>.
On 6/2/07, Rajith Attapattu <ra...@gmail.com> wrote:Azeez>>
> Similarly, there is a cleanup mechanism which runs whenever a new contexts
> gets added. Hence there is no need to send out a removeContext message.
>
Rajith>>
This is not good enough. We need to remove a context ASAP as soon as a
> client logs out. Consider this. Client logs into Node A and does some work.
> All information is replicated to Node B & C. The client logout from Node A.
> We don't replicate the remove command and no new contexts are added, hence
> no cleanup mechanism is executed.
> Now some other entity can point to Node B or C and continue the session as
> those contexts are still available in those nodes. Further any cleanup
> associated with the lifecycle of the session (ex resource release) will not
> be executed on Node B and C. These kind of situations will make the cluster
> unreliable.
>
> We can't compromise integrity for the sake of performance. We need to
> achieve both without compromising each other.
>
>
What about the situation where a session has been created on behalf of a
client and this session is replicated, and the session is still active? A
malicious entity can get hold of the session id and connect to any node. In
Axis2, there are 2 types of sessions; soapsession & transportsession. In the
case of the soapsession, the session is tracked using the service group
context id in the addressing header. If proper security is not used, anybody
can capture this id and masquerade as the legitimate client. Similary, in
transportsession, the SessionContext is associated with the session
management mechanism of the transport. Even in such a case, if proper
security at the transport level is not used, a masquerading attack can take
place. My point is, removing the session from the nodes is not the solution
for securing the system. We must use WS-Security to properly handle this
requirement.
--
Thanks
Afkham Azeez
http://www.wso2.org
GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Rajith Attapattu <ra...@gmail.com>.
On 6/1/07, Afkham Azeez <af...@gmail.com> wrote:
>
>
>
> On 5/31/07, Rajith Attapattu <rajith77@gmail.com > wrote:
> >
> > I introduced distinct commands (add/update/remove) to make it more
> > simple and readable (to avoid update context being too complicated).
>
>
> Please take a look at UpdateServiceGroupContextCommand.java or any other
> Update context command and let me know of any code that looks complicated.
>
I Azeez didn't imply that the current update code was complicated or for
that matter any reference to the current code. My comment was about my
thinking behind the design decision to introduce distinct events instead of
one.
When there are thousands of requests, the NW will get flooded with
> clustering messages. We have to minimize the number of messages we send.
>
I agree with optimizing the messages we send across the network as much as
possible. So +1 for that. Having distinct commands doesn't mean u need to
send them seperately. You can pack the create and update commands together
(or create a single command to for this), bcos a create is always followed
by an update. So optimisations can be achived independent of the way we
handle our events.
When I commited the initial code drop it was more of a prototype for proof
of concept and was more focused on getting it working rather than
performance (which was my second goal), however my bad for not keeping an
eye on performance.
>This also simplifies the clustering implementation, since an updateContext
> > message can be received before a contextAdded message due to the order in
> > which the >threads run at a particular time.
> > Afkam, I didn't encounter this during my very limited testing. However
> > an update is always followed after a context is created. I am not sure how
> > these events get mixed up due to thread issues. We should be able to
> > guarantee causal ordering at the least.
>
>
> This might not have occurred in a simple unit testing scenario, but does
> occur under some load.
>
I tested the initial implementation on 5 nodes, however I never load tested
:(.
The ReplicationHandler sends out the Update context message before the
> create context message is sent out by the ContextManager.
>
This is certainly worth investigating. As I said we need to garuntee causal
ordering at the least.
I'm investigating this. However, I strongly feel that we need not
> unnecessarily send messages. Do you see any advantage in sending an
> addContext message when a single updateContext message can do the trick?
> Also note that I've changed the ReplicationHandler to send out only a single
> updateContext message which aggregates all the updates to all the the
> relevant contexts, instead of sending a message for each property in each
> context. Please review the new architecture & code.
>
I don't see an advantage in sending more messages than we have to (see my
comment above). However replicating distinct events can be an advantage.
When I receive an update how would I know whether it was a create+update or
a regular update ?
The absence of a the context in the receiving node doesn't necessarily imply
that it was create + update. It could be that this node was down when the
real creation happened.
If we have logic which is sensitive to the life cycle of a service, it
might be important to know distinct events. However as I mentioned
distinct events can be batched (I'd rather use the word packed) together
as opposed to be sent as different messages thereby achieving the
optimisation u noted.
Similarly, there is a cleanup mechanism which runs whenever a new contexts
> gets added. Hence there is no need to send out a removeContext message.
>
This is not good enough. We need to remove a context ASAP as soon as a
client logs out. Consider this. Client logs into Node A and does some work.
All information is replicated to Node B & C. The client logout from Node A.
We don't replicate the remove command and no new contexts are added, hence
no cleanup mechanism is executed.
Now some other entity can point to Node B or C and continue the session as
those contexts are still available in those nodes. Further any cleanup
associated with the lifecycle of the session (ex resource release) will not
be executed on Node B and C. These kind of situations will make the cluster
unreliable.
We can't compromise integrity for the sake of performance. We need to
achieve both without compromising each other.
IMHO, these changes will greatly reduce the complexity and improve the
> efficiency of the clustering implementation.
>
> Can you explain the problem a bit more? To me the solution seems more of a
> > hack due to some other issues in the system.
>
>
> There may be some problems which need investigation, but this definitely
> is not a hack. Think of a node receiving a request. At that point, the
> relevant contexts are created. In other words, contexts are created on
> demand. Similarly, what is wrong in creating contexts on demand only when
> necessary, when an update message is received? Why send unnecessary cleanup
> messages when Axis2 already has a mechanism to do this?
>
See my comment about cleanup .
Anyways if u think this change makes it easy, please go ahead with it.
> >
> > Regards,
> >
> > Rajith
> >
> > On 5/30/07, Chamikara Jayalath < chamikaramj@gmail.com> wrote:
> > >
> > > Hi Azeez,
> > >
> > > On 5/30/07, Afkham Azeez < afkham@gmail.com> wrote:
> > > >
> > > > I think there is no need for the contextAdded & contextRemoved
> > > > methods in the ContextManager interface. contextUpdated should suffice.
> > > >
> > > > The reason is, when a contextUpdated message is received, if it does
> > > > not exist on a particular node, it can be created. Similary, the context
> > > > cleanup mechanism will take care of cleaning up the contexts on each node,
> > > > hence removeContext is also not needed. This also simplifies the clustering
> > > > implementation, since an updateContext message can be received before a
> > > > contextAdded message due to the order in which the threads run at a
> > > > particular time. Otherwise, the sender has to ensure that always a
> > > > createContext message is sent to the channel before an updateContext message
> > > > is sent.
> > > >
> > > > Thoughts?
> > >
> > >
> > > Agree. +1 for the change.
> > >
> > > Chamikara
> > >
> > >
> > > --
> > > > Thanks
> > > > Afkham Azeez
> > > >
> > > > http://www.wso2.org
> > > > GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
> > >
> > >
> > >
> > >
> > > --
> > > Chamikara Jayalath
> > > WSO2 Inc.
> > > http://wso2.com/
> > > http://wso2.org/ - For your Oxygen needs
> >
> >
> >
>
>
> --
> Thanks
> Afkham Azeez
>
> http://www.wso2.org
> GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
>
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Afkham Azeez <af...@gmail.com>.
On 5/31/07, Rajith Attapattu <ra...@gmail.com> wrote:
>
> I introduced distinct commands (add/update/remove) to make it more simple
> and readable (to avoid update context being too complicated).
Please take a look at UpdateServiceGroupContextCommand.java or any other
Update context command and let me know of any code that looks complicated.
When there are thousands of requests, the NW will get flooded with
clustering messages. We have to minimize the number of messages we send.
>This also simplifies the clustering implementation, since an updateContext
> message can be received before a contextAdded message due to the order in
> which the >threads run at a particular time.
> Afkam, I didn't encounter this during my very limited testing. However an
> update is always followed after a context is created. I am not sure how
> these events get mixed up due to thread issues. We should be able to
> guarantee causal ordering at the least.
This might not have occurred in a simple unit testing scenario, but does
occur under some load. The ReplicationHandler sends out the Update context
message before the create context message is sent out by the ContextManager.
I'm investigating this. However, I strongly feel that we need not
unnecessarily send messages. Do you see any advantage in sending an
addContext message when a single updateContext message can do the trick?
Also note that I've changed the ReplicationHandler to send out only a single
updateContext message which aggregates all the updates to all the the
relevant contexts, instead of sending a message for each property in each
context. Please review the new architecture & code.
Similarly, there is a cleanup mechanism which runs whenever a new contexts
gets added. Hence there is no need to send out a removeContext message.
IMHO, these changes will greatly reduce the complexity and improve the
efficiency of the clustering implementation.
Can you explain the problem a bit more? To me the solution seems more of a
> hack due to some other issues in the system.
There may be some problems which need investigation, but this definitely
is not a hack. Think of a node receiving a request. At that point, the
relevant contexts are created. In other words, contexts are created on
demand. Similarly, what is wrong in creating contexts on demand only when
necessary, when an update message is received? Why send unnecessary cleanup
messages when Axis2 already has a mechanism to do this?
Anyways if u think this change makes it easy, please go ahead with it.
>
> Regards,
>
> Rajith
>
> On 5/30/07, Chamikara Jayalath < chamikaramj@gmail.com> wrote:
> >
> > Hi Azeez,
> >
> > On 5/30/07, Afkham Azeez < afkham@gmail.com> wrote:
> > >
> > > I think there is no need for the contextAdded & contextRemoved methods
> > > in the ContextManager interface. contextUpdated should suffice.
> > >
> > > The reason is, when a contextUpdated message is received, if it does
> > > not exist on a particular node, it can be created. Similary, the context
> > > cleanup mechanism will take care of cleaning up the contexts on each node,
> > > hence removeContext is also not needed. This also simplifies the clustering
> > > implementation, since an updateContext message can be received before a
> > > contextAdded message due to the order in which the threads run at a
> > > particular time. Otherwise, the sender has to ensure that always a
> > > createContext message is sent to the channel before an updateContext message
> > > is sent.
> > >
> > > Thoughts?
> >
> >
> > Agree. +1 for the change.
> >
> > Chamikara
> >
> >
> > --
> > > Thanks
> > > Afkham Azeez
> > >
> > > http://www.wso2.org
> > > GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
> >
> >
> >
> >
> > --
> > Chamikara Jayalath
> > WSO2 Inc.
> > http://wso2.com/
> > http://wso2.org/ - For your Oxygen needs
>
>
>
--
Thanks
Afkham Azeez
http://www.wso2.org
GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Rajith Attapattu <ra...@gmail.com>.
I introduced distinct commands (add/update/remove) to make it more simple
and readable (to avoid update context being too complicated).
>This also simplifies the clustering implementation, since an updateContext
message can be received before a contextAdded message due to the order in
which the >threads run at a particular time.
Afkam, I didn't encounter this during my very limited testing. However an
update is always followed after a context is created. I am not sure how
these events get mixed up due to thread issues. We should be able to
garuntee causal ordering at the least.
Can you explain the problem a bit more? To me the solution seems more of a
hack due to some other issues in the system.
Anyways if u think this change makes it easy, please go ahead with it.
Regards,
Rajith
On 5/30/07, Chamikara Jayalath <ch...@gmail.com> wrote:
>
> Hi Azeez,
>
> On 5/30/07, Afkham Azeez <af...@gmail.com> wrote:
> >
> > I think there is no need for the contextAdded & contextRemoved methods
> > in the ContextManager interface. contextUpdated should suffice.
> >
> > The reason is, when a contextUpdated message is received, if it does not
> > exist on a particular node, it can be created. Similary, the context cleanup
> > mechanism will take care of cleaning up the contexts on each node, hence
> > removeContext is also not needed. This also simplifies the clustering
> > implementation, since an updateContext message can be received before a
> > contextAdded message due to the order in which the threads run at a
> > particular time. Otherwise, the sender has to ensure that always a
> > createContext message is sent to the channel before an updateContext message
> > is sent.
> >
> > Thoughts?
>
>
> Agree. +1 for the change.
>
> Chamikara
>
>
> --
> > Thanks
> > Afkham Azeez
> >
> > http://www.wso2.org
> > GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
>
>
>
>
> --
> Chamikara Jayalath
> WSO2 Inc.
> http://wso2.com/
> http://wso2.org/ - For your Oxygen needs
Re: [Axis2] Clustering: No need for ContextManager contextAdded & contextRemoved methods
Posted by Chamikara Jayalath <ch...@gmail.com>.
Hi Azeez,
On 5/30/07, Afkham Azeez <af...@gmail.com> wrote:
>
> I think there is no need for the contextAdded & contextRemoved methods in
> the ContextManager interface. contextUpdated should suffice.
>
> The reason is, when a contextUpdated message is received, if it does not
> exist on a particular node, it can be created. Similary, the context cleanup
> mechanism will take care of cleaning up the contexts on each node, hence
> removeContext is also not needed. This also simplifies the clustering
> implementation, since an updateContext message can be received before a
> contextAdded message due to the order in which the threads run at a
> particular time. Otherwise, the sender has to ensure that always a
> createContext message is sent to the channel before an updateContext message
> is sent.
>
> Thoughts?
Agree. +1 for the change.
Chamikara
--
> Thanks
> Afkham Azeez
>
> http://www.wso2.org
> GPG Fingerprint: 643F C2AF EB78 F886 40C9 B2A2 4AE2 C887 665E 0760
--
Chamikara Jayalath
WSO2 Inc.
http://wso2.com/
http://wso2.org/ - For your Oxygen needs