You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yang <te...@gmail.com> on 2011/06/25 04:58:03 UTC
network/message simulation framework?
I'd like to verify the behavior of Cassandra under some edge case message
loss scenarios.
it's rather difficult to reproduce such things, cuz you have to setup
multiple servers, and on each box essentially control
the message "gates" to any other nodes in the network. the realistic way
that I can think of is to close off all traffic and only manually
allow certain messages to pass through. (in the case of Cassandra, we need
to let through all gossip messages, and manually control
replication messages )
are there some existing simulation frameworks around? junit + various mock
frameworks look like a good fit, but
in reality is not enough to simulate complex network code, for example, it's
basically impossible to setup a simulated
N-node cluster on only one test box.
Thanks
Yang
Re: network/message simulation framework?
Posted by Yang <te...@gmail.com>.
I see now that you can obtain "cheap" multiple IPs on the same box by
creating virtual NICs, and then create one thrift server on each of these
NIC/IPs
but a remaining problem is that to do the tests, you'd have to fire up
multiple JVMs, because a lot of structures in Cassandra are static, then the
tests
can be carried out in a shell script organizing several java processes,
slightly less convenient than a single junit process which orchestrates
those
multiple servers in a single JVM
yang
On Sat, Jun 25, 2011 at 3:40 AM, Watanabe Maki <wa...@gmail.com>wrote:
> iptables?
>
> From iPhone
>
>
> On 2011/06/25, at 12:47, Yang <te...@gmail.com> wrote:
>
> Thanks Jonathan.
>
> this provides a way to essentially get a copy of the outgoing messages,
> the messages onto the real connections still go through, but I would need a
> way
> to shut off the real connections too.
>
> shutting off the connections could probably done by mocking the
> TCPconnection class,
> but an even more difficult thing is that the ThriftSocket etc will need to
> open a java.net.Socket, which
> hooks onto the real IP address and port, in a simulation environment, we'd
> need to
> mock out the java.net.Socket too, that sounds more difficult.
>
> yang
>
> On Fri, Jun 24, 2011 at 8:23 PM, Jonathan Ellis < <jb...@gmail.com>
> jbellis@gmail.com> wrote:
>
>> The MessageSink code is designed for this. Look in
>> MessagingService.sendOneWay.
>>
>> On Fri, Jun 24, 2011 at 9:58 PM, Yang < <te...@gmail.com>
>> teddyyyy123@gmail.com> wrote:
>> > I'd like to verify the behavior of Cassandra under some edge case
>> message
>> > loss scenarios.
>> > it's rather difficult to reproduce such things, cuz you have to setup
>> > multiple servers, and on each box essentially control
>> > the message "gates" to any other nodes in the network. the realistic way
>> > that I can think of is to close off all traffic and only manually
>> > allow certain messages to pass through. (in the case of Cassandra, we
>> need
>> > to let through all gossip messages, and manually control
>> > replication messages )
>> > are there some existing simulation frameworks around? junit + various
>> mock
>> > frameworks look like a good fit, but
>> > in reality is not enough to simulate complex network code, for example,
>> it's
>> > basically impossible to setup a simulated
>> > N-node cluster on only one test box.
>> >
>> > Thanks
>> > Yang
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> <http://www.datastax.com>http://www.datastax.com
>>
>
>
Re: network/message simulation framework?
Posted by Watanabe Maki <wa...@gmail.com>.
iptables?
From iPhone
On 2011/06/25, at 12:47, Yang <te...@gmail.com> wrote:
> Thanks Jonathan.
>
> this provides a way to essentially get a copy of the outgoing messages,
> the messages onto the real connections still go through, but I would need a way
> to shut off the real connections too.
>
> shutting off the connections could probably done by mocking the TCPconnection class,
> but an even more difficult thing is that the ThriftSocket etc will need to open a java.net.Socket, which
> hooks onto the real IP address and port, in a simulation environment, we'd need to
> mock out the java.net.Socket too, that sounds more difficult.
>
> yang
>
> On Fri, Jun 24, 2011 at 8:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> The MessageSink code is designed for this. Look in MessagingService.sendOneWay.
>
> On Fri, Jun 24, 2011 at 9:58 PM, Yang <te...@gmail.com> wrote:
> > I'd like to verify the behavior of Cassandra under some edge case message
> > loss scenarios.
> > it's rather difficult to reproduce such things, cuz you have to setup
> > multiple servers, and on each box essentially control
> > the message "gates" to any other nodes in the network. the realistic way
> > that I can think of is to close off all traffic and only manually
> > allow certain messages to pass through. (in the case of Cassandra, we need
> > to let through all gossip messages, and manually control
> > replication messages )
> > are there some existing simulation frameworks around? junit + various mock
> > frameworks look like a good fit, but
> > in reality is not enough to simulate complex network code, for example, it's
> > basically impossible to setup a simulated
> > N-node cluster on only one test box.
> >
> > Thanks
> > Yang
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: network/message simulation framework?
Posted by Jonathan Ellis <jb...@gmail.com>.
On Fri, Jun 24, 2011 at 10:47 PM, Yang <te...@gmail.com> wrote:
> Thanks Jonathan.
> this provides a way to essentially get a copy of the outgoing messages,
> the messages onto the real connections still go through, but I would need a
> way
> to shut off the real connections too.
Just return null from the sink to disappear the messages.
> but an even more difficult thing is that the ThriftSocket etc will need to
> open a java.net.Socket, which
> hooks onto the real IP address and port, in a simulation environment, we'd
> need to
> mock out the java.net.Socket too, that sounds more difficult.
Yes, this won't help on the Thrift side.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: network/message simulation framework?
Posted by Yang <te...@gmail.com>.
never mind, I see it does work for stopping messages. but still can't
simulate multiple IPs on one box.
On Fri, Jun 24, 2011 at 8:47 PM, Yang <te...@gmail.com> wrote:
> Thanks Jonathan.
>
> this provides a way to essentially get a copy of the outgoing messages,
> the messages onto the real connections still go through, but I would need a
> way
> to shut off the real connections too.
>
> shutting off the connections could probably done by mocking the
> TCPconnection class,
> but an even more difficult thing is that the ThriftSocket etc will need to
> open a java.net.Socket, which
> hooks onto the real IP address and port, in a simulation environment, we'd
> need to
> mock out the java.net.Socket too, that sounds more difficult.
>
> yang
>
>
> On Fri, Jun 24, 2011 at 8:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> The MessageSink code is designed for this. Look in
>> MessagingService.sendOneWay.
>>
>> On Fri, Jun 24, 2011 at 9:58 PM, Yang <te...@gmail.com> wrote:
>> > I'd like to verify the behavior of Cassandra under some edge case
>> message
>> > loss scenarios.
>> > it's rather difficult to reproduce such things, cuz you have to setup
>> > multiple servers, and on each box essentially control
>> > the message "gates" to any other nodes in the network. the realistic way
>> > that I can think of is to close off all traffic and only manually
>> > allow certain messages to pass through. (in the case of Cassandra, we
>> need
>> > to let through all gossip messages, and manually control
>> > replication messages )
>> > are there some existing simulation frameworks around? junit + various
>> mock
>> > frameworks look like a good fit, but
>> > in reality is not enough to simulate complex network code, for example,
>> it's
>> > basically impossible to setup a simulated
>> > N-node cluster on only one test box.
>> >
>> > Thanks
>> > Yang
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
Re: network/message simulation framework?
Posted by Yang <te...@gmail.com>.
Thanks Jonathan.
this provides a way to essentially get a copy of the outgoing messages,
the messages onto the real connections still go through, but I would need a
way
to shut off the real connections too.
shutting off the connections could probably done by mocking the
TCPconnection class,
but an even more difficult thing is that the ThriftSocket etc will need to
open a java.net.Socket, which
hooks onto the real IP address and port, in a simulation environment, we'd
need to
mock out the java.net.Socket too, that sounds more difficult.
yang
On Fri, Jun 24, 2011 at 8:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> The MessageSink code is designed for this. Look in
> MessagingService.sendOneWay.
>
> On Fri, Jun 24, 2011 at 9:58 PM, Yang <te...@gmail.com> wrote:
> > I'd like to verify the behavior of Cassandra under some edge case message
> > loss scenarios.
> > it's rather difficult to reproduce such things, cuz you have to setup
> > multiple servers, and on each box essentially control
> > the message "gates" to any other nodes in the network. the realistic way
> > that I can think of is to close off all traffic and only manually
> > allow certain messages to pass through. (in the case of Cassandra, we
> need
> > to let through all gossip messages, and manually control
> > replication messages )
> > are there some existing simulation frameworks around? junit + various
> mock
> > frameworks look like a good fit, but
> > in reality is not enough to simulate complex network code, for example,
> it's
> > basically impossible to setup a simulated
> > N-node cluster on only one test box.
> >
> > Thanks
> > Yang
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: network/message simulation framework?
Posted by Jonathan Ellis <jb...@gmail.com>.
The MessageSink code is designed for this. Look in MessagingService.sendOneWay.
On Fri, Jun 24, 2011 at 9:58 PM, Yang <te...@gmail.com> wrote:
> I'd like to verify the behavior of Cassandra under some edge case message
> loss scenarios.
> it's rather difficult to reproduce such things, cuz you have to setup
> multiple servers, and on each box essentially control
> the message "gates" to any other nodes in the network. the realistic way
> that I can think of is to close off all traffic and only manually
> allow certain messages to pass through. (in the case of Cassandra, we need
> to let through all gossip messages, and manually control
> replication messages )
> are there some existing simulation frameworks around? junit + various mock
> frameworks look like a good fit, but
> in reality is not enough to simulate complex network code, for example, it's
> basically impossible to setup a simulated
> N-node cluster on only one test box.
>
> Thanks
> Yang
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com