You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Noel J. Bergman" <no...@devtech.com> on 2002/10/18 20:25:37 UTC
Testing: Plans, Scripts and Tools
I have looked at some of the recent posts, and get the sense that the test
scripts are ad hoc. It seems to me that we ought to adopt a more formal
methodology, develop a test strategy, and base the initial majority of
scripts on testing for compliance with the RFCs. Additional scripts can be
based upon specific needs, e.g., accommodating real-world compatibility with
specific applications. We should have documented scripts that explain what
the standard is, what any special deviation(s) might be, what is being
tested, and why.
For these "telnet-based" protocols, I believe that we can (and should)
leverage regular expressions, rather than just plain canned strings for
comparing. That way we can better handle response strings, and use data
from matched groups within our following commands. I am also of a mind that
we should look at using BSF as a key component of our testing framework.
That would let us leverage simple templates as well as more complex scripts,
including those adapted from other suites. Harmeet mentioned Python, which
can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html.
Done properly, this becomes a very useful general purpose test bed,
customized for a given protocol by additional beans and scripts.
Lastly, we needn't have an NIH approach to this sort of external testing. I
am sure that we can find other tests to supplement those that we build.
Thoughts?
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Testing: Plans, Scripts and Tools
Posted by Harmeet Bedi <ha...@kodemuse.com>.
From: "Nathan Carter" <na...@gmx.net>
> 2) jpcap is a java wrapper for libpcap/tcpdump that would allow us to
> capture sessions from both Windows and *nix machines.
Never played with this, but sounds very interesting.
> I'm thinking that I could use 2) to build a recorder to create the (C:
> blah S:blah) test scripts that we're currently using. Does this sound
> helpful to others?
I have been using Outlook Express troubleshooting log files to build the
pop3/smtp scripts (Tools/Options/Maintenance troubleshooting checkboxs)
Emacs-gnus can record also all commands to a buffer.
These settings in .emacs should do the trick.
(setq nntp-record-commands t)
(setq smtpmail-debug-info t)
(setq pop3-debug t)
A tcpdump record could be cool and useful. One advantage would be to see
what other client and server are sending around. At some point I think it
would be very useful reverse engineer and build an exchange protocol
replacement.
A general strategy would be good. Another alternative but possibly not as
general could be to have a NAT type recording proxy. Basically record and
forward connections.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Testing: Plans, Scripts and Tools
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I think we should have some replay scripts or test to
> verify smtp auth extension, size extension and open
> relay test.
SMTP AUTH, SIZE extension and other RFC related tests should be labeled with
the RFC section being tested and the expected dialog. So if someone says
that James isn't properly implementing RFC XXXX, section aa.bb.cc, we can
verify the RFC against the test script generated from the RFC, and James'
behavior. We should not be building RFC test scripts by recording them.
The only scripts that should be build by recording them are those expressing
non-standard "real-world" behavior, where responding the same way as the
recorded session is necessary.
> It would be cool to point users to a test in response to questions like:
> Does James support SMTP Auth?
THAT script should be written from the RFC, whereas
> Is my James server configured to be open relay?
is really a suite of tests that can be adopted from real-world Open Relay
testers. Keep in mind that such test scripts can't verify anything with the
current approach. They can affect the SMTP protocol, but James rejects
relaying attempts asynchronously in the processor chain. We'd need another
means to verify that James hasn't relayed the e-mail. There are several
options for that step.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Testing: Plans, Scripts and Tools
Posted by Harmeet Bedi <ha...@kodemuse.com>.
----- Original Message -----
From: "Serge Knystautas" <se...@lokitech.com>
To: "James Developers List" <ja...@jakarta.apache.org>
Sent: Friday, October 18, 2002 10:04 PM
Subject: Re: Testing: Plans, Scripts and Tools
> Well, James can just log all the communication (you shouldn't need some
> separate TCP analysis).
For James specific stuff, James logs and or client sessions would be very
simple to do and immediatly provide good validation.
TCP analysis level recording may be useful for say seeing what xyz server
does and compare. It could also help reverse engineer and build protocols
that we haven't got yet. For instance wouldn't it be cool if we build
exchange protocol handler. :-)
> relatively small number of possible clients that you need to handle. SMTP
> has a lot more implementation variations... again though, just keep your
Agreed that SMTP has huge variants. I think we should have some replay
scripts or test to verify smtp auth extension, size extension and open relay
test.
It would be cool to point users to a test in response to questions like:
Does James support SMTP Auth ? Is my James server configured to be open
relay ?
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Testing: Plans, Scripts and Tools
Posted by Nathan Carter <na...@gmx.net>.
Noel J. Bergman wrote:
> I don't disagree with the need for real-world client compliance tests. For
> example, I can tell you that we have an issue with Yahoo! web mail. It uses
> an obsolete (deprecated) POP command. My point is that each test should
> have a defined role within the test plan. The RFCs are one coverage area
> that must be addressed, and real-world applications are another. And I
> disagree with the idea of generating test script for RFC dictated behavior
> by recording a live session. That just documents how that particular
> implementation implemented the RFC.
>
+1. Clearly both RFC compliance and real-world compliance need to be
integrated into into the more formal testing framework that you propose.
> Each James handler logs all protocol traffic at the DEBUG log level. That
> is how I know about the problem with Yahoo! web mail, although I'm not
> really keen to implement an obsolete command just for them. The place for
> using a packet sniffer or proxy would be where we want to understand some
> extension, such as one that might be found between Outlook and Exchange. A
> modified version of tcpdump that can be set to filter on a particular
> IP:port would be a good solution on supported platforms, since it would
> require no changes of any kind. Another proxy option would be the TCP GUI
> monitor program that is packaged with Axis. It acts as a local proxy,
> recording the traffic and displaying each side of the protocol in a split
> window. It requires configuring the local side of the protocol to work with
> the proxy, bit it does work on any Java enabled platform, and with any
> application.
>
Didn't know about the Axis option, though I've used a similar two-sided
GUI.
> Hopefully I am expressing myself clearly.
>
> --- Noel
>
> -----Original Message-----
> From: Serge Knystautas [mailto:sergek@lokitech.com]
> Sent: Saturday, October 19, 2002 1:05
> To: James Developers List
> Subject: Re: Testing: Plans, Scripts and Tools
>
>
> Well, James can just log all the communication (you shouldn't need some
> separate TCP analysis). If a transcript works, you extract the session and
> add it as a test to always check moving forward. If it fais, well, we've
> got something to fix in James and then set it as a test. :)
>
> For my money, I think the hard part is not POP3 and IMAP... there is a
> relatively small number of possible clients that you need to handle. SMTP
> has a lot more implementation variations... again though, just keep your
> James server at debug level and grab those sessions for test suites.
>
> Serge Knystautas
> Loki Technologies
> http://www.lokitech.com/
>
> ----- Original Message -----
> From: "Nathan Carter" <na...@gmx.net>
> To: "James Developers List" <ja...@jakarta.apache.org>
> Sent: Saturday, October 19, 2002 12:39 AM
> Subject: Re: Testing: Plans, Scripts and Tools
>
>
>
>>Noel,
>>
>>In *addition to* the "RFC-compliance" tests, it seems necessary to me
>>(especially looking forward to IMAP, where real-world client compliance
>>may not be as strict as with POP3) to have some sort of
>>"session-recorder" that would make recording new scripts from real-world
>>clients almost trivial. I've been looking at building such a recorder
>>using a modified packet sniffer:
>>1) tcpflow is an extension to tcpdump that writes data portion of all
>>filtered packets to file. (*nix only)
>>2) jpcap is a java wrapper for libpcap/tcpdump that would allow us to
>>capture sessions from both Windows and *nix machines.
>>
>>I'm thinking that I could use 2) to build a recorder to create the (C:
>>blah S:blah) test scripts that we're currently using. Does this sound
>>helpful to others?
>>
>>Jmeter has a similar "recording" feature built on a simple local proxy,
>>but it's a bit painful to turn the proxy on/off and change your browser
>>settings every time you want to record. Thoughts?
>>
>>
>>Nathan
>>
>>
>>
>>Noel J. Bergman wrote:
>>
>>>I have looked at some of the recent posts, and get the sense that the te
>>
> st
>
>>>scripts are ad hoc. It seems to me that we ought to adopt a more formal
>>>methodology, develop a test strategy, and base the initial majority of
>>>scripts on testing for compliance with the RFCs. Additional scripts can
>>
> be
>
>>>based upon specific needs, e.g., accommodating real-world compatibility
>>
> with
>
>>>specific applications. We should have documented scripts that explain
>>
> what
>
>>>the standard is, what any special deviation(s) might be, what is being
>>>tested, and why.
>>>
>>>For these "telnet-based" protocols, I believe that we can (and should)
>>>leverage regular expressions, rather than just plain canned strings for
>>>comparing. That way we can better handle response strings, and use data
>>>from matched groups within our following commands. I am also of a mind
>>
> that
>
>>>we should look at using BSF as a key component of our testing framework.
>>>That would let us leverage simple templates as well as more complex
>>
> scripts,
>
>>>including those adapted from other suites. Harmeet mentioned Python,
>>
> which
>
>>>can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html.
>>>Done properly, this becomes a very useful general purpose test bed,
>>>customized for a given protocol by additional beans and scripts.
>>>
>>>Lastly, we needn't have an NIH approach to this sort of external
>>
> testing. I
>
>>>am sure that we can find other tests to supplement those that we build.
>>>
>>>Thoughts?
>>>
>>>--- Noel
>>
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
-Nathan
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Testing: Plans, Scripts and Tools
Posted by "Noel J. Bergman" <no...@devtech.com>.
I don't disagree with the need for real-world client compliance tests. For
example, I can tell you that we have an issue with Yahoo! web mail. It uses
an obsolete (deprecated) POP command. My point is that each test should
have a defined role within the test plan. The RFCs are one coverage area
that must be addressed, and real-world applications are another. And I
disagree with the idea of generating test script for RFC dictated behavior
by recording a live session. That just documents how that particular
implementation implemented the RFC.
Each James handler logs all protocol traffic at the DEBUG log level. That
is how I know about the problem with Yahoo! web mail, although I'm not
really keen to implement an obsolete command just for them. The place for
using a packet sniffer or proxy would be where we want to understand some
extension, such as one that might be found between Outlook and Exchange. A
modified version of tcpdump that can be set to filter on a particular
IP:port would be a good solution on supported platforms, since it would
require no changes of any kind. Another proxy option would be the TCP GUI
monitor program that is packaged with Axis. It acts as a local proxy,
recording the traffic and displaying each side of the protocol in a split
window. It requires configuring the local side of the protocol to work with
the proxy, bit it does work on any Java enabled platform, and with any
application.
Hopefully I am expressing myself clearly.
--- Noel
-----Original Message-----
From: Serge Knystautas [mailto:sergek@lokitech.com]
Sent: Saturday, October 19, 2002 1:05
To: James Developers List
Subject: Re: Testing: Plans, Scripts and Tools
Well, James can just log all the communication (you shouldn't need some
separate TCP analysis). If a transcript works, you extract the session and
add it as a test to always check moving forward. If it fais, well, we've
got something to fix in James and then set it as a test. :)
For my money, I think the hard part is not POP3 and IMAP... there is a
relatively small number of possible clients that you need to handle. SMTP
has a lot more implementation variations... again though, just keep your
James server at debug level and grab those sessions for test suites.
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
----- Original Message -----
From: "Nathan Carter" <na...@gmx.net>
To: "James Developers List" <ja...@jakarta.apache.org>
Sent: Saturday, October 19, 2002 12:39 AM
Subject: Re: Testing: Plans, Scripts and Tools
> Noel,
>
> In *addition to* the "RFC-compliance" tests, it seems necessary to me
> (especially looking forward to IMAP, where real-world client compliance
> may not be as strict as with POP3) to have some sort of
> "session-recorder" that would make recording new scripts from real-world
> clients almost trivial. I've been looking at building such a recorder
> using a modified packet sniffer:
> 1) tcpflow is an extension to tcpdump that writes data portion of all
> filtered packets to file. (*nix only)
> 2) jpcap is a java wrapper for libpcap/tcpdump that would allow us to
> capture sessions from both Windows and *nix machines.
>
> I'm thinking that I could use 2) to build a recorder to create the (C:
> blah S:blah) test scripts that we're currently using. Does this sound
> helpful to others?
>
> Jmeter has a similar "recording" feature built on a simple local proxy,
> but it's a bit painful to turn the proxy on/off and change your browser
> settings every time you want to record. Thoughts?
>
>
> Nathan
>
>
>
> Noel J. Bergman wrote:
> > I have looked at some of the recent posts, and get the sense that the te
st
> > scripts are ad hoc. It seems to me that we ought to adopt a more formal
> > methodology, develop a test strategy, and base the initial majority of
> > scripts on testing for compliance with the RFCs. Additional scripts can
be
> > based upon specific needs, e.g., accommodating real-world compatibility
with
> > specific applications. We should have documented scripts that explain
what
> > the standard is, what any special deviation(s) might be, what is being
> > tested, and why.
> >
> > For these "telnet-based" protocols, I believe that we can (and should)
> > leverage regular expressions, rather than just plain canned strings for
> > comparing. That way we can better handle response strings, and use data
> > from matched groups within our following commands. I am also of a mind
that
> > we should look at using BSF as a key component of our testing framework.
> > That would let us leverage simple templates as well as more complex
scripts,
> > including those adapted from other suites. Harmeet mentioned Python,
which
> > can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html.
> > Done properly, this becomes a very useful general purpose test bed,
> > customized for a given protocol by additional beans and scripts.
> >
> > Lastly, we needn't have an NIH approach to this sort of external
testing. I
> > am sure that we can find other tests to supplement those that we build.
> >
> > Thoughts?
> >
> > --- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Testing: Plans, Scripts and Tools
Posted by Serge Knystautas <se...@lokitech.com>.
Well, James can just log all the communication (you shouldn't need some
separate TCP analysis). If a transcript works, you extract the session and
add it as a test to always check moving forward. If it fais, well, we've
got something to fix in James and then set it as a test. :)
For my money, I think the hard part is not POP3 and IMAP... there is a
relatively small number of possible clients that you need to handle. SMTP
has a lot more implementation variations... again though, just keep your
James server at debug level and grab those sessions for test suites.
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
----- Original Message -----
From: "Nathan Carter" <na...@gmx.net>
To: "James Developers List" <ja...@jakarta.apache.org>
Sent: Saturday, October 19, 2002 12:39 AM
Subject: Re: Testing: Plans, Scripts and Tools
> Noel,
>
> In *addition to* the "RFC-compliance" tests, it seems necessary to me
> (especially looking forward to IMAP, where real-world client compliance
> may not be as strict as with POP3) to have some sort of
> "session-recorder" that would make recording new scripts from real-world
> clients almost trivial. I've been looking at building such a recorder
> using a modified packet sniffer:
> 1) tcpflow is an extension to tcpdump that writes data portion of all
> filtered packets to file. (*nix only)
> 2) jpcap is a java wrapper for libpcap/tcpdump that would allow us to
> capture sessions from both Windows and *nix machines.
>
> I'm thinking that I could use 2) to build a recorder to create the (C:
> blah S:blah) test scripts that we're currently using. Does this sound
> helpful to others?
>
> Jmeter has a similar "recording" feature built on a simple local proxy,
> but it's a bit painful to turn the proxy on/off and change your browser
> settings every time you want to record. Thoughts?
>
>
> Nathan
>
>
>
> Noel J. Bergman wrote:
> > I have looked at some of the recent posts, and get the sense that the te
st
> > scripts are ad hoc. It seems to me that we ought to adopt a more formal
> > methodology, develop a test strategy, and base the initial majority of
> > scripts on testing for compliance with the RFCs. Additional scripts can
be
> > based upon specific needs, e.g., accommodating real-world compatibility
with
> > specific applications. We should have documented scripts that explain
what
> > the standard is, what any special deviation(s) might be, what is being
> > tested, and why.
> >
> > For these "telnet-based" protocols, I believe that we can (and should)
> > leverage regular expressions, rather than just plain canned strings for
> > comparing. That way we can better handle response strings, and use data
> > from matched groups within our following commands. I am also of a mind
that
> > we should look at using BSF as a key component of our testing framework.
> > That would let us leverage simple templates as well as more complex
scripts,
> > including those adapted from other suites. Harmeet mentioned Python,
which
> > can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html.
> > Done properly, this becomes a very useful general purpose test bed,
> > customized for a given protocol by additional beans and scripts.
> >
> > Lastly, we needn't have an NIH approach to this sort of external
testing. I
> > am sure that we can find other tests to supplement those that we build.
> >
> > Thoughts?
> >
> > --- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Testing: Plans, Scripts and Tools
Posted by Nathan Carter <na...@gmx.net>.
Noel,
In *addition to* the "RFC-compliance" tests, it seems necessary to me
(especially looking forward to IMAP, where real-world client compliance
may not be as strict as with POP3) to have some sort of
"session-recorder" that would make recording new scripts from real-world
clients almost trivial. I've been looking at building such a recorder
using a modified packet sniffer:
1) tcpflow is an extension to tcpdump that writes data portion of all
filtered packets to file. (*nix only)
2) jpcap is a java wrapper for libpcap/tcpdump that would allow us to
capture sessions from both Windows and *nix machines.
I'm thinking that I could use 2) to build a recorder to create the (C:
blah S:blah) test scripts that we're currently using. Does this sound
helpful to others?
Jmeter has a similar "recording" feature built on a simple local proxy,
but it's a bit painful to turn the proxy on/off and change your browser
settings every time you want to record. Thoughts?
Nathan
Noel J. Bergman wrote:
> I have looked at some of the recent posts, and get the sense that the test
> scripts are ad hoc. It seems to me that we ought to adopt a more formal
> methodology, develop a test strategy, and base the initial majority of
> scripts on testing for compliance with the RFCs. Additional scripts can be
> based upon specific needs, e.g., accommodating real-world compatibility with
> specific applications. We should have documented scripts that explain what
> the standard is, what any special deviation(s) might be, what is being
> tested, and why.
>
> For these "telnet-based" protocols, I believe that we can (and should)
> leverage regular expressions, rather than just plain canned strings for
> comparing. That way we can better handle response strings, and use data
> from matched groups within our following commands. I am also of a mind that
> we should look at using BSF as a key component of our testing framework.
> That would let us leverage simple templates as well as more complex scripts,
> including those adapted from other suites. Harmeet mentioned Python, which
> can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html.
> Done properly, this becomes a very useful general purpose test bed,
> customized for a given protocol by additional beans and scripts.
>
> Lastly, we needn't have an NIH approach to this sort of external testing. I
> am sure that we can find other tests to supplement those that we build.
>
> Thoughts?
>
> --- Noel
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Testing: Plans, Scripts and Tools
Posted by Harmeet Bedi <ha...@kodemuse.com>.
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
> It seems to me that we ought to adopt a more formal
> methodology, develop a test strategy, and base the initial majority of
> scripts on testing for compliance with the RFCs.
Agreed, we should talk methodology and right set of tools.
Here are some ideas on methodology.
- Create protocol compliance verification tests using record replay scripts
- Create Test suites to for end to end testing. This may involve multiple
protocol tests.
- Create tests to expose and verify specific problems. For instance Danny's
end to end test crashes James unless proposed scheduler implemenation or
watchdog build is used. There could be simulate thread pool problem where
threads were not being retuned back.
I have been trying to build some tools that will facilitate testing.
These are the 2 types of general tools.
- Protocol Simulation given a template file
- JUnit Test Sequence and test suite assembly. It could be used to do
something like send a mail, wait for mail to show up via POP3 or IMAP by
writing separate tests for SMTP/POP3/IMAP and combining them together in a
declarative manner.
> I have looked at some of the recent posts, and get the sense that the test
> scripts are ad hoc.
Test tools etc. are a bit ad hoc. I thought about various alternatives but
it seemed easiest to write something like ProtocolSimulator. It seemed to
simple and none of the programs I know of seem to fit the bill.
One good thing is that we have some tests and tools and have now started
talking more about testing.
> I am also of a mind that
> we should look at using BSF as a key component of our testing framework.
> That would let us leverage simple templates as well as more complex
scripts,
> including those adapted from other suites. Harmeet mentioned Python,
which
> can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html.
> Done properly, this becomes a very useful general purpose test bed,
> customized for a given protocol by additional beans and scripts.
I think Jython is really useful.
- It support the Python libraries. Good client libraries for protocols like
POP3, SMTP, IMAP, NNTP etc. For example commons does not seem to have IMAP
library. Writing non record-replay tests may be painful.
- Easy to extend.
- Very cool language and tool. Much easier to work with than Java.
- Seamless mapping and integration with Java language. Most other Java
languages like TCL, etc. have more convoluted mapping.
One downside to any scripting language is the lack of strong typing.
Compiling the code can be useful esp. if tests once created and working may
not be touched for a long time at least not all code paths.
It may be simpler and lesser learning curve to directly use Jython instead
of Jython+BSF but either alternative works for me.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>