You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Zach Richardson <j....@gmail.com> on 2011/10/21 23:28:23 UTC
Trying to write tests for CASSANDRA-3127 (Internode Compression)
Hi All,
I have been working on an implementation for internode compression
(CASSANDRA-3127.) https://issues.apache.org/jira/browse/CASSANDRA-3127
I have written code that "works," but I'm looking for some advice on
how to write unit tests for it. At the moment it compresses where:
interrnode_message_compression_threshold > 0 means it compress
messages larger than it
compresses all messags if it is == 0
and compresses none if less than 0
The code itself has been tested in an environment outside of cassandra
(i.e. a few mock classes, and a heavily modified OutboundTcpConnection
and IncomingTcpConnection.) and inside of Cassandra all of the current
unit tests are passing.
Since I can't inject a mocked MessagingService into
IncomingTcpConnection, I'm guessing I have to do the testing from the
outside of MessagingService, but the MessagingService itself checks to
see if you are sending messages to yourself, and doesn't put them over
the connection. Are there any tricks to letting me mock different
endpoints from within a unit test?
Can this only be tested in a distributed fashion at the OS level?
Also the internode_message_compression_threshold is set through the
cassandra.yaml file--is it possible to set these properties at
runtime? Can I just change the public entry for it in the static
Config class directly, or will that break other things?
Thanks for you help and time,
Zach
Re: Trying to write tests for CASSANDRA-3127 (Internode Compression)
Posted by Zach Richardson <j....@gmail.com>.
Nick,
Thanks for the reply!
I ended up creating a custom IVerbHandler to receive the messages from
MessagingService, and when I created messages, I set their from field
to an address != localhost, so they did not get optimized. This
appeared to work!
Thanks again.
Zach
On Mon, Oct 24, 2011 at 10:25 AM, Nick Bailey <ni...@datastax.com> wrote:
> Not sure if you saw this, but org.apache.cassandra.net.sink has a
> SinkManager class and an interface for implementing message sinks. Basically
> it lets you catch messages as they are being sent or received by
> MessagingService.
>
> Could be useful, and is used in a couple other tests.
>
> On Sun, Oct 23, 2011 at 11:54 AM, Zach Richardson <za...@raveldata.com>wrote:
>
>> Thanks,
>>
>> Will give these a shot.
>>
>> On Sun, Oct 23, 2011 at 11:18 AM, Jonathan Ellis <jb...@gmail.com>
>> wrote:
>>
>> > I see a couple options.
>> >
>> > StorageProxy has this constant:
>> >
>> > private static final boolean OPTIMIZE_LOCAL_REQUESTS = true; //
>> > set to false to test messagingservice path on single node
>> >
>> > So, you could make it an instance variable and create a SP object with
>> > it set to false for tests.
>> >
>> > Or, you could do a test using ccm for multinode control, as in the
>> > long_read.sh test on
>> > https://issues.apache.org/jira/browse/CASSANDRA-3303.
>> >
>> > On Fri, Oct 21, 2011 at 4:28 PM, Zach Richardson
>> > <j....@gmail.com> wrote:
>> > > Hi All,
>> > >
>> > > I have been working on an implementation for internode compression
>> > > (CASSANDRA-3127.) https://issues.apache.org/jira/browse/CASSANDRA-3127
>> > >
>> > > I have written code that "works," but I'm looking for some advice on
>> > > how to write unit tests for it. At the moment it compresses where:
>> > >
>> > > interrnode_message_compression_threshold > 0 means it compress
>> > > messages larger than it
>> > > compresses all messags if it is == 0
>> > > and compresses none if less than 0
>> > >
>> > > The code itself has been tested in an environment outside of cassandra
>> > > (i.e. a few mock classes, and a heavily modified OutboundTcpConnection
>> > > and IncomingTcpConnection.) and inside of Cassandra all of the current
>> > > unit tests are passing.
>> > >
>> > > Since I can't inject a mocked MessagingService into
>> > > IncomingTcpConnection, I'm guessing I have to do the testing from the
>> > > outside of MessagingService, but the MessagingService itself checks to
>> > > see if you are sending messages to yourself, and doesn't put them over
>> > > the connection. Are there any tricks to letting me mock different
>> > > endpoints from within a unit test?
>> > >
>> > > Can this only be tested in a distributed fashion at the OS level?
>> > >
>> > > Also the internode_message_compression_threshold is set through the
>> > > cassandra.yaml file--is it possible to set these properties at
>> > > runtime? Can I just change the public entry for it in the static
>> > > Config class directly, or will that break other things?
>> > >
>> > > Thanks for you help and time,
>> > >
>> > > Zach
>> > >
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>> >
>>
>>
>>
>> --
>> Zach Richardson
>> Ravel, Co-founder
>> Austin, TX
>> zach@raveldata.com
>> 512.825.6031
>>
>
Re: Trying to write tests for CASSANDRA-3127 (Internode Compression)
Posted by Nick Bailey <ni...@datastax.com>.
Not sure if you saw this, but org.apache.cassandra.net.sink has a
SinkManager class and an interface for implementing message sinks. Basically
it lets you catch messages as they are being sent or received by
MessagingService.
Could be useful, and is used in a couple other tests.
On Sun, Oct 23, 2011 at 11:54 AM, Zach Richardson <za...@raveldata.com>wrote:
> Thanks,
>
> Will give these a shot.
>
> On Sun, Oct 23, 2011 at 11:18 AM, Jonathan Ellis <jb...@gmail.com>
> wrote:
>
> > I see a couple options.
> >
> > StorageProxy has this constant:
> >
> > private static final boolean OPTIMIZE_LOCAL_REQUESTS = true; //
> > set to false to test messagingservice path on single node
> >
> > So, you could make it an instance variable and create a SP object with
> > it set to false for tests.
> >
> > Or, you could do a test using ccm for multinode control, as in the
> > long_read.sh test on
> > https://issues.apache.org/jira/browse/CASSANDRA-3303.
> >
> > On Fri, Oct 21, 2011 at 4:28 PM, Zach Richardson
> > <j....@gmail.com> wrote:
> > > Hi All,
> > >
> > > I have been working on an implementation for internode compression
> > > (CASSANDRA-3127.) https://issues.apache.org/jira/browse/CASSANDRA-3127
> > >
> > > I have written code that "works," but I'm looking for some advice on
> > > how to write unit tests for it. At the moment it compresses where:
> > >
> > > interrnode_message_compression_threshold > 0 means it compress
> > > messages larger than it
> > > compresses all messags if it is == 0
> > > and compresses none if less than 0
> > >
> > > The code itself has been tested in an environment outside of cassandra
> > > (i.e. a few mock classes, and a heavily modified OutboundTcpConnection
> > > and IncomingTcpConnection.) and inside of Cassandra all of the current
> > > unit tests are passing.
> > >
> > > Since I can't inject a mocked MessagingService into
> > > IncomingTcpConnection, I'm guessing I have to do the testing from the
> > > outside of MessagingService, but the MessagingService itself checks to
> > > see if you are sending messages to yourself, and doesn't put them over
> > > the connection. Are there any tricks to letting me mock different
> > > endpoints from within a unit test?
> > >
> > > Can this only be tested in a distributed fashion at the OS level?
> > >
> > > Also the internode_message_compression_threshold is set through the
> > > cassandra.yaml file--is it possible to set these properties at
> > > runtime? Can I just change the public entry for it in the static
> > > Config class directly, or will that break other things?
> > >
> > > Thanks for you help and time,
> > >
> > > Zach
> > >
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
>
>
>
> --
> Zach Richardson
> Ravel, Co-founder
> Austin, TX
> zach@raveldata.com
> 512.825.6031
>
Re: Trying to write tests for CASSANDRA-3127 (Internode Compression)
Posted by Zach Richardson <za...@raveldata.com>.
Thanks,
Will give these a shot.
On Sun, Oct 23, 2011 at 11:18 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> I see a couple options.
>
> StorageProxy has this constant:
>
> private static final boolean OPTIMIZE_LOCAL_REQUESTS = true; //
> set to false to test messagingservice path on single node
>
> So, you could make it an instance variable and create a SP object with
> it set to false for tests.
>
> Or, you could do a test using ccm for multinode control, as in the
> long_read.sh test on
> https://issues.apache.org/jira/browse/CASSANDRA-3303.
>
> On Fri, Oct 21, 2011 at 4:28 PM, Zach Richardson
> <j....@gmail.com> wrote:
> > Hi All,
> >
> > I have been working on an implementation for internode compression
> > (CASSANDRA-3127.) https://issues.apache.org/jira/browse/CASSANDRA-3127
> >
> > I have written code that "works," but I'm looking for some advice on
> > how to write unit tests for it. At the moment it compresses where:
> >
> > interrnode_message_compression_threshold > 0 means it compress
> > messages larger than it
> > compresses all messags if it is == 0
> > and compresses none if less than 0
> >
> > The code itself has been tested in an environment outside of cassandra
> > (i.e. a few mock classes, and a heavily modified OutboundTcpConnection
> > and IncomingTcpConnection.) and inside of Cassandra all of the current
> > unit tests are passing.
> >
> > Since I can't inject a mocked MessagingService into
> > IncomingTcpConnection, I'm guessing I have to do the testing from the
> > outside of MessagingService, but the MessagingService itself checks to
> > see if you are sending messages to yourself, and doesn't put them over
> > the connection. Are there any tricks to letting me mock different
> > endpoints from within a unit test?
> >
> > Can this only be tested in a distributed fashion at the OS level?
> >
> > Also the internode_message_compression_threshold is set through the
> > cassandra.yaml file--is it possible to set these properties at
> > runtime? Can I just change the public entry for it in the static
> > Config class directly, or will that break other things?
> >
> > Thanks for you help and time,
> >
> > Zach
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
--
Zach Richardson
Ravel, Co-founder
Austin, TX
zach@raveldata.com
512.825.6031
Re: Trying to write tests for CASSANDRA-3127 (Internode Compression)
Posted by Jonathan Ellis <jb...@gmail.com>.
I see a couple options.
StorageProxy has this constant:
private static final boolean OPTIMIZE_LOCAL_REQUESTS = true; //
set to false to test messagingservice path on single node
So, you could make it an instance variable and create a SP object with
it set to false for tests.
Or, you could do a test using ccm for multinode control, as in the
long_read.sh test on
https://issues.apache.org/jira/browse/CASSANDRA-3303.
On Fri, Oct 21, 2011 at 4:28 PM, Zach Richardson
<j....@gmail.com> wrote:
> Hi All,
>
> I have been working on an implementation for internode compression
> (CASSANDRA-3127.) https://issues.apache.org/jira/browse/CASSANDRA-3127
>
> I have written code that "works," but I'm looking for some advice on
> how to write unit tests for it. At the moment it compresses where:
>
> interrnode_message_compression_threshold > 0 means it compress
> messages larger than it
> compresses all messags if it is == 0
> and compresses none if less than 0
>
> The code itself has been tested in an environment outside of cassandra
> (i.e. a few mock classes, and a heavily modified OutboundTcpConnection
> and IncomingTcpConnection.) and inside of Cassandra all of the current
> unit tests are passing.
>
> Since I can't inject a mocked MessagingService into
> IncomingTcpConnection, I'm guessing I have to do the testing from the
> outside of MessagingService, but the MessagingService itself checks to
> see if you are sending messages to yourself, and doesn't put them over
> the connection. Are there any tricks to letting me mock different
> endpoints from within a unit test?
>
> Can this only be tested in a distributed fashion at the OS level?
>
> Also the internode_message_compression_threshold is set through the
> cassandra.yaml file--is it possible to set these properties at
> runtime? Can I just change the public entry for it in the static
> Config class directly, or will that break other things?
>
> Thanks for you help and time,
>
> Zach
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com