You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Alex Karasulu <ak...@apache.org> on 2009/04/01 14:49:51 UTC
[GSoC] Protocol performance testing framework
Hi all,
Please excuse the cross post but this proposal for a Google Summer of Code
project applies as much to Apache MINA as it does to Apache Directory. The
idea is very simple, you have the following components:
(1) A MINA protocol server that is being tested.
(2) A MINA IoFilter for statistics collection for use in any MINA protocol
server. This filter may collect information from the system, the JVM and
from MINA itself. It can sample this information and push this information
into the statistics collector (#4 below). It will probably expose some out
of band socket for communcation and control.
(3) A load injecting agent which can accept PDU's or a sequence of PDU's
to inject into the MINA protocol server. Several of these may be
instantiated on different hosts to generate load on the server. The PDU's
are prepared and supplied in advance before begining a test from the agent
control UI (#4). These agents have a socket channel for command and control
from the Agent Control UI. They will all be started together to start
generating load against the server. PDU's are supplied to keep these agents
ignorant about the protocol being used. This way the control UI can be used
to prepare PDU's in advance.
(4) Agent control UI and statistics collator. This is the command center
that communicates with the agents to tell them what PDUs to send to which
endpoints and at what frequence if things cycle. This agent also gathers
statistics from all the agents to get client latency information as well as
to generate overall throughput statistics from the time collated data
obtained from all the load inject agents/clients. It also controls the
sampling rate of the IoFilter injected into the server and collects data
from it to correlate memory usage, CPU, server side latencies and other
factors effecting server performance.
We at Directory were thinking of building something like this for LDAP but
then quickly realized that we also need this for DNS, DHCP, Kerberos, and
ChangePassword and anything else we may snap into the server. We're sure
any other protocol could benefit from this as well. So it might be a good
idea to combine efforts while giving the many GSoC students we've had
responses from a chance to do something more at other Apache Projects. Plus
we have a few RCP developers at Directory one of whom was willing to lend a
hand to help build the basis for the agent control UI and statistics
collector.
So if students and others are interested in this I'm sure something could be
built by the end of summer that will be useful to anyone using MINA.
Regards,
Alex
Re: [GSoC] Protocol performance testing framework
Posted by Ashish <pa...@gmail.com>.
>> May be STAF can be a solution. Have not used it
>
> This is interesting. I was not aware of this framework (damn thing snuck by
> me). I'll take a look at this but I know for sure that the correlation
> between what's going on inside the MINA server via stat collection, the
> host, through the MINA filter along with the clients is not something this
> framework can do even if we expose it via JMX.
Agreed, this tool is just a framework. To put in a couple of
requirements for this solution
1. Shall be able to generate protocol packets
2. Shall be able to emulate Agent to generate load of packet packets
3. Shall be able to execute Agents on distributed setup (multiple machines/OS)
4. Shall be able to collect Server Stats (in a configurable way, so
that we can go beyond network layer)
5. Shall be able to correlate Server Stats with Agents logs
6. Shall be able to generate report out of the data
7. Shall be able to control/configure/monitor all running Agents
Some more thought
If we can implement the same as Eclipse plugin, would be great.
Have one more recommendation as a tool JSystemTest
(http://www.jsystemtest.org/).
The tool is based on Eclipse, though it brings in its own philosophy,
but being open source we can extends it.
This tool is on my top priority list to be evaluated for testing
automation. Have interacted with these folks, and found them quite
serious about their offerings.
I see a good use of ProtoTiger
(http://www.tabbyinstall.com/prototiger/ProtoTiger.html) to be used
for Packet code generation.
thanks
Re: [GSoC] Protocol performance testing framework
Posted by Ashish <pa...@gmail.com>.
>> May be STAF can be a solution. Have not used it
>
> This is interesting. I was not aware of this framework (damn thing snuck by
> me). I'll take a look at this but I know for sure that the correlation
> between what's going on inside the MINA server via stat collection, the
> host, through the MINA filter along with the clients is not something this
> framework can do even if we expose it via JMX.
Agreed, this tool is just a framework. To put in a couple of
requirements for this solution
1. Shall be able to generate protocol packets
2. Shall be able to emulate Agent to generate load of packet packets
3. Shall be able to execute Agents on distributed setup (multiple machines/OS)
4. Shall be able to collect Server Stats (in a configurable way, so
that we can go beyond network layer)
5. Shall be able to correlate Server Stats with Agents logs
6. Shall be able to generate report out of the data
7. Shall be able to control/configure/monitor all running Agents
Some more thought
If we can implement the same as Eclipse plugin, would be great.
Have one more recommendation as a tool JSystemTest
(http://www.jsystemtest.org/).
The tool is based on Eclipse, though it brings in its own philosophy,
but being open source we can extends it.
This tool is on my top priority list to be evaluated for testing
automation. Have interacted with these folks, and found them quite
serious about their offerings.
I see a good use of ProtoTiger
(http://www.tabbyinstall.com/prototiger/ProtoTiger.html) to be used
for Packet code generation.
thanks
Re: [GSoC] Protocol performance testing framework
Posted by Alex Karasulu <ak...@gmail.com>.
On Wed, Apr 1, 2009 at 3:18 PM, Ashish <pa...@gmail.com> wrote:
>
> May be STAF can be a solution. Have not used it
>
This is interesting. I was not aware of this framework (damn thing snuck by
me). I'll take a look at this but I know for sure that the correlation
between what's going on inside the MINA server via stat collection, the
host, through the MINA filter along with the clients is not something this
framework can do even if we expose it via JMX.
The idea is to collect information from within the server, from the clients
producing the load and collate it all based on time. Then use this to
assemble a good picture of what bottlenecks exist. Combined with a profiler
like Yourkit this could produce some very interesting results.
Alex
Re: [GSoC] Protocol performance testing framework
Posted by Ashish <pa...@gmail.com>.
> Please excuse the cross post but this proposal for a Google Summer of Code
> project applies as much to Apache MINA as it does to Apache Directory. The
> idea is very simple, you have the following components:
>
> (1) A MINA protocol server that is being tested.
> (2) A MINA IoFilter for statistics collection for use in any MINA protocol
> server. This filter may collect information from the system, the JVM and
> from MINA itself. It can sample this information and push this information
> into the statistics collector (#4 below). It will probably expose some out
> of band socket for communcation and control.
We can use JMX to collect stats.
> (3) A load injecting agent which can accept PDU's or a sequence of PDU's
> to inject into the MINA protocol server. Several of these may be
> instantiated on different hosts to generate load on the server. The PDU's
> are prepared and supplied in advance before begining a test from the agent
> control UI (#4). These agents have a socket channel for command and control
> from the Agent Control UI. They will all be started together to start
> generating load against the server. PDU's are supplied to keep these agents
> ignorant about the protocol being used. This way the control UI can be used
> to prepare PDU's in advance.
Couple of options here
http://staf.sourceforge.net/ -
>From the web site
"The Software Testing Automation Framework (STAF) is an open source,
multi-platform, multi-language framework designed around the idea of
reusable components, called services (such as process invocation,
resource management, logging, and monitoring). STAF removes the tedium
of building an automation infrastructure, thus enabling you to focus
on building your automation solution. The STAF framework provides the
foundation upon which to build higher level solutions, and provides a
pluggable approach supported across a large variety of platforms and
languages."
> (4) Agent control UI and statistics collator. This is the command center
> that communicates with the agents to tell them what PDUs to send to which
> endpoints and at what frequence if things cycle. This agent also gathers
> statistics from all the agents to get client latency information as well as
> to generate overall throughput statistics from the time collated data
> obtained from all the load inject agents/clients. It also controls the
> sampling rate of the IoFilter injected into the server and collects data
> from it to correlate memory usage, CPU, server side latencies and other
> factors effecting server performance.
May be STAF can be a solution. Have not used it
> We at Directory were thinking of building something like this for LDAP but
> then quickly realized that we also need this for DNS, DHCP, Kerberos, and
> ChangePassword and anything else we may snap into the server. We're sure
> any other protocol could benefit from this as well. So it might be a good
> idea to combine efforts while giving the many GSoC students we've had
> responses from a chance to do something more at other Apache Projects. Plus
> we have a few RCP developers at Directory one of whom was willing to lend a
> hand to help build the basis for the agent control UI and statistics
> collector.
>
> So if students and others are interested in this I'm sure something could be
> built by the end of summer that will be useful to anyone using MINA.
>
thanks
ashish
Re: [GSoC] Protocol performance testing framework
Posted by Eero Nevalainen <ee...@indagon.com>.
Alex Karasulu wrote:
> Hi all,
>
> Please excuse the cross post but this proposal for a Google Summer of Code
> project applies as much to Apache MINA as it does to Apache Directory. The
> idea is very simple, you have the following components:
>
> (1) A MINA protocol server that is being tested.
> (2) A MINA IoFilter for statistics collection for use in any MINA protocol
> server. This filter may collect information from the system, the JVM and
> from MINA itself. It can sample this information and push this information
> into the statistics collector (#4 below). It will probably expose some out
> of band socket for communcation and control.
> (3) A load injecting agent which can accept PDU's or a sequence of PDU's
> to inject into the MINA protocol server. Several of these may be
> instantiated on different hosts to generate load on the server. The PDU's
> are prepared and supplied in advance before begining a test from the agent
> control UI (#4). These agents have a socket channel for command and control
> from the Agent Control UI. They will all be started together to start
> generating load against the server. PDU's are supplied to keep these agents
> ignorant about the protocol being used. This way the control UI can be used
> to prepare PDU's in advance.
> (4) Agent control UI and statistics collator. This is the command center
> that communicates with the agents to tell them what PDUs to send to which
> endpoints and at what frequence if things cycle. This agent also gathers
> statistics from all the agents to get client latency information as well as
> to generate overall throughput statistics from the time collated data
> obtained from all the load inject agents/clients. It also controls the
> sampling rate of the IoFilter injected into the server and collects data
> from it to correlate memory usage, CPU, server side latencies and other
> factors effecting server performance.
Is there some specific reason for not using the Grinder?
http://grinder.sourceforge.net/index.html
I used it a couple of years ago and was pleased with it. The only thing
we needed to add before getting to actual testing was some shell scripts
to start the agents over SSH.
--
Eero Nevalainen
Re: [GSoC] Protocol performance testing framework
Posted by Ashish <pa...@gmail.com>.
> Please excuse the cross post but this proposal for a Google Summer of Code
> project applies as much to Apache MINA as it does to Apache Directory. The
> idea is very simple, you have the following components:
>
> (1) A MINA protocol server that is being tested.
> (2) A MINA IoFilter for statistics collection for use in any MINA protocol
> server. This filter may collect information from the system, the JVM and
> from MINA itself. It can sample this information and push this information
> into the statistics collector (#4 below). It will probably expose some out
> of band socket for communcation and control.
We can use JMX to collect stats.
> (3) A load injecting agent which can accept PDU's or a sequence of PDU's
> to inject into the MINA protocol server. Several of these may be
> instantiated on different hosts to generate load on the server. The PDU's
> are prepared and supplied in advance before begining a test from the agent
> control UI (#4). These agents have a socket channel for command and control
> from the Agent Control UI. They will all be started together to start
> generating load against the server. PDU's are supplied to keep these agents
> ignorant about the protocol being used. This way the control UI can be used
> to prepare PDU's in advance.
Couple of options here
http://staf.sourceforge.net/ -
>From the web site
"The Software Testing Automation Framework (STAF) is an open source,
multi-platform, multi-language framework designed around the idea of
reusable components, called services (such as process invocation,
resource management, logging, and monitoring). STAF removes the tedium
of building an automation infrastructure, thus enabling you to focus
on building your automation solution. The STAF framework provides the
foundation upon which to build higher level solutions, and provides a
pluggable approach supported across a large variety of platforms and
languages."
> (4) Agent control UI and statistics collator. This is the command center
> that communicates with the agents to tell them what PDUs to send to which
> endpoints and at what frequence if things cycle. This agent also gathers
> statistics from all the agents to get client latency information as well as
> to generate overall throughput statistics from the time collated data
> obtained from all the load inject agents/clients. It also controls the
> sampling rate of the IoFilter injected into the server and collects data
> from it to correlate memory usage, CPU, server side latencies and other
> factors effecting server performance.
May be STAF can be a solution. Have not used it
> We at Directory were thinking of building something like this for LDAP but
> then quickly realized that we also need this for DNS, DHCP, Kerberos, and
> ChangePassword and anything else we may snap into the server. We're sure
> any other protocol could benefit from this as well. So it might be a good
> idea to combine efforts while giving the many GSoC students we've had
> responses from a chance to do something more at other Apache Projects. Plus
> we have a few RCP developers at Directory one of whom was willing to lend a
> hand to help build the basis for the agent control UI and statistics
> collector.
>
> So if students and others are interested in this I'm sure something could be
> built by the end of summer that will be useful to anyone using MINA.
>
thanks
ashish