Vysper unit tests on running server

[Niklas Gustavsson <ni...@protocol7.com>]

Hey

Over at FtpServer, we run a lot of unit tests but starting the server
and then letting a client (in our case commons-net) interact with it,
testing for correct outcomes. I'm thinking something similar would be
a good idea for Vysper but fail to find any such tests (besides
org.apache.vysper.smack.BasicClient but it looks more like something
which is run interactively during development). What do you think?

/niklas

Re: Vysper unit tests on running server

[Niklas Gustavsson <ni...@protocol7.com>]

On Tue, Jun 23, 2009 at 8:39 AM, Bernd Fondermann<bf...@brainlounge.de> wrote:
> I kicked this off because Niklas plans to implement the Ping XEP and I
> assumed he's looking for a way to write proper tests for it. And I just
> wanted to point out that there might be good ways to do that without
> needing a fully-fledged running server and scripting a client.

I'm doing both for the ping code, just to get a feeling for how to
best write the client driven tests. Will take a few more days since
I'm kind of busy at work at the moment.

/niklas

Re: Vysper unit tests on running server

[Bernd Fondermann <bf...@brainlounge.de>]

Michael Jakl wrote:
> Hi!
> 
> On Mon, Jun 22, 2009 at 22:59, Bernd
> Fondermann<be...@googlemail.com> wrote:
>> On Mon, Jun 22, 2009 at 13:48, Michael Jakl<ja...@gmail.com> wrote:
>>> Since the clients run under the same hood, checking whether a message
>>> arrives within a certain amount of time should be easy. With more than
>>> one client-implementation the chances are high that we catch
>>> interoperability bugs earlier.
>> +1 for the general idea. -0 for doing this. Vysper is about a spec
>> conforming server implementation, not about clients.
>> If there are clients with interop problems, well, just fix *them*
>> first (but let's still be liberal in what the server accepts, within
>> the scope of the spec).
>> If the server has a problem, write a test for that.
> 
> "Interoperability bug" was not exactly the correct wording. The goal
> was to identify server bugs by using "proven" clients. I'm also -0 for
> doing this right now, but it seems like a cheap way for doing
> spec-conformance tests (if there are no better ways).

+1. Ok, now I get it. Currently, our use of Smack is not very
systematic. We can improve this, but I'd stick to one client lib. Smack
is ok for me.

I kicked this off because Niklas plans to implement the Ping XEP and I
assumed he's looking for a way to write proper tests for it. And I just
wanted to point out that there might be good ways to do that without
needing a fully-fledged running server and scripting a client.

  Bernd

Re: Vysper unit tests on running server

[Michael Jakl <ja...@gmail.com>]

Hi!

On Mon, Jun 22, 2009 at 22:59, Bernd
Fondermann<be...@googlemail.com> wrote:
> On Mon, Jun 22, 2009 at 13:48, Michael Jakl<ja...@gmail.com> wrote:
>> Since the clients run under the same hood, checking whether a message
>> arrives within a certain amount of time should be easy. With more than
>> one client-implementation the chances are high that we catch
>> interoperability bugs earlier.
>
> +1 for the general idea. -0 for doing this. Vysper is about a spec
> conforming server implementation, not about clients.
> If there are clients with interop problems, well, just fix *them*
> first (but let's still be liberal in what the server accepts, within
> the scope of the spec).
> If the server has a problem, write a test for that.

"Interoperability bug" was not exactly the correct wording. The goal
was to identify server bugs by using "proven" clients. I'm also -0 for
doing this right now, but it seems like a cheap way for doing
spec-conformance tests (if there are no better ways).

Michael

Re: Vysper unit tests on running server

[Bernd Fondermann <be...@googlemail.com>]

On Mon, Jun 22, 2009 at 13:48, Michael Jakl<ja...@gmail.com> wrote:
> Hi!
>
> On Sun, Jun 21, 2009 at 22:48, Niklas Gustavsson<ni...@protocol7.com> wrote:
>> On Sun, Jun 21, 2009 at 10:21 PM, Bernd Fondermann<bf...@brainlounge.de> wrote:
>>> But we're kind of talking very theoretical here (otherwise my answer
>>> could have been shorter and there'd be somebody still listening down
>>> here). Let's talk about a concrete example.
>>
>> Agreed, I'll see what I can come up with :-)
>
> What about defining a high-level test suite which is abstract enough
> to be carried out by more than one XMPP library? I'm thinking of an
> abstracted API which can run on Smack and other libs for
> - openeing a connection,
> - registering useres,
> - sending messages and the like.

This goes beyond the scope of this server implementation.

> Since the clients run under the same hood, checking whether a message
> arrives within a certain amount of time should be easy. With more than
> one client-implementation the chances are high that we catch
> interoperability bugs earlier.

+1 for the general idea. -0 for doing this. Vysper is about a spec
conforming server implementation, not about clients.
If there are clients with interop problems, well, just fix *them*
first (but let's still be liberal in what the server accepts, within
the scope of the spec).
If the server has a problem, write a test for that.

Just as a side note: there are no guarantees for delivery/quality of
service/realtime requirements in the XMPP spec.
if the server decides to delay a message (because he has 100K stanzas
in the queue) he is perfectly entitled to do so.

> For example if I want to test my pubsub implementation I'd add 3
> users, one publisher and two subscribers. When publishing a message,
> the two subscribers should receive a notification within a certain
> amount of time (some seconds), if not we've got a problem.

So you want to test the general stanza delivery. go ahead and do so!
this is worthwhile.
but this is nothing the pubsub implementation should be concerned
about. pubsub tests should only test pubsub specifics.

> This, of course, is complementary to the unit (whitebox) tests.
>
> Just a few early thoughts on this.

  Bernd

Re: Vysper unit tests on running server

[Michael Jakl <ja...@gmail.com>]

Hi!

On Sun, Jun 21, 2009 at 22:48, Niklas Gustavsson<ni...@protocol7.com> wrote:
> On Sun, Jun 21, 2009 at 10:21 PM, Bernd Fondermann<bf...@brainlounge.de> wrote:
>> But we're kind of talking very theoretical here (otherwise my answer
>> could have been shorter and there'd be somebody still listening down
>> here). Let's talk about a concrete example.
>
> Agreed, I'll see what I can come up with :-)

What about defining a high-level test suite which is abstract enough
to be carried out by more than one XMPP library? I'm thinking of an
abstracted API which can run on Smack and other libs for
- openeing a connection,
- registering useres,
- sending messages and the like.

Since the clients run under the same hood, checking whether a message
arrives within a certain amount of time should be easy. With more than
one client-implementation the chances are high that we catch
interoperability bugs earlier.

For example if I want to test my pubsub implementation I'd add 3
users, one publisher and two subscribers. When publishing a message,
the two subscribers should receive a notification within a certain
amount of time (some seconds), if not we've got a problem.

This, of course, is complementary to the unit (whitebox) tests.

Just a few early thoughts on this.

Michael

Re: Vysper unit tests on running server

[Niklas Gustavsson <ni...@protocol7.com>]

On Sun, Jun 21, 2009 at 10:21 PM, Bernd Fondermann<bf...@brainlounge.de> wrote:

[snipped some good arguments]

> XMPP is not only specified for socket transport. It's defined BOSH (over
> HTTP) as well. I'm sure you wouldn't want to double every e2e test for
> both transports?

Yeah, I've worked a lot with XMPP (or Jabber in those days, for those
with historical interests, here's a protocol proposal I wrote in 2001,
ages ago http://www.protocol7.com/jabber/whiteboard_proposal.txt) over
HTTP earlier (using it from browsers). While I don't think its a bad
idea to have tests running over both HTTP and TCP, it might not be
worth the double effort (if that what it would take).

>> Again, let me point out that these tests should never replace the
>> current tests but merely test at a different abstraction.
>
> Yes, I agree this could prove useful. But in most scenarios I can think
> of at the moment, more unit-like tests would be
> + simpler
> + easier to write and maintain
> + running much faster
> + succeed more often ;-)

Agreeing with all but the last. If these tests would turn out to be
unstable, we should throw them out.

> But we're kind of talking very theoretical here (otherwise my answer
> could have been shorter and there'd be somebody still listening down
> here). Let's talk about a concrete example.

Agreed, I'll see what I can come up with :-)

/niklas

Re: Vysper unit tests on running server

[Bernd Fondermann <bf...@brainlounge.de>]

Niklas Gustavsson wrote:
> On Sun, Jun 21, 2009 at 11:14 AM, Bernd Fondermann<bf...@brainlounge.de> wrote:
>> For testing new functionality (mostly handlers), starting the server is
>> not neccessary (and that's a big plus). Just testing the handler in
>> isolation is usually sufficient.
> 
> I should have pointed out that I did not suggest to in any way replace
> the current testing (nor step away from the design that allows
> handlers and other classes to be tested in isloation), that should
> still be done!

And I didn't understand it like you were suggesting this.

>> At some point, we might need that integration tests, to make sure
>> everything works properly together, though.
> 
> Yeah, that's kind of the experience we've had with FtpServer. The
> integration tests has located tons of bugs, and allows us easily to
> write tests for cases reported by users ("server does this when my
> client does that").
> 
>> What do you intend to use this end-to-end testing for?
> 
> I would say that we should intend to test most, if not all, of the
> client facing functionality this way (as well as with more
> fine-grained test on each class).

Mmhh. don't know. I agree with the "users reported odd behavior"
approach. I agree to test those stuff that can go wrong on a socket with
a real socket.

But keep in mind: XMPP has a pretty expensive initial handshaking protocol.

>> Integration tests for XMPP may prove difficult, because
>> + XMPP is XML and the way elements are closed or allowance of arbitrary
>> whitespaces can add for a lot of noise, elements order may be arbitrary etc.
>> + stanzas may come out of order, too
>> + extensions are manyfold
>> + tracking session state is easier with our own testing facilities
> 
> I probably (surely) don't understand all the details here. But, my
> idea would be to use Smack as the client which at least should isolate
> us as much as we want from the protocol (but still allow us to create
> arbitrary packages for testing odd cases). I also think this would
> handle out of order stanzas and extensions. Not sure about session
> state.

You could be wrong here. There are no out-of-order stanzas in XMPP which
you'd have to test for. This is completely ok for this protocol. But
receiving and sending things *will* make end-to-end tests harder to succeed.

Some stanzas, though, are prohibited/valid at certain session states.
There is a separate layer taking care of this (see ProtocolWorker and
SessionState). So there is no need to test this on a socket, because the
socket does not add anything useful to the test (except for odd cases
maybe, but not in general).

Session states are very important in XMPP. Ca. 50% of RFC3920 deals with
the states the session goes through on initial handshake.
(see http://tools.ietf.org/html/draft-saintandre-rfc3920bis-09 chapters
4, 5, 6, 7 and 8)
So, Vysper uses a state machine to track this. Which works independent
of a socket. It should just work for any stream of stanzas and is
testable on this level. See for example all subclasses of
AbstractProtocolStateTestCase.

XMPP is different to line-oriented protocols like FTP, SMTP, Telnet etc.
XMPP is using a subset of XML, so it is less strict about syntax (the
set of wellformedness is larger).
<iq id="ab1" result="cancel" />
is semantically the same as
<iq
  result='cancel'
  iq='ab1'
>
</iq>

That's why I saw very early on that it's very painful to test against
the character stream. Now I prefer to test everything that deals with
stanzas against Stanza objects.

XMPP is not only specified for socket transport. It's defined BOSH (over
HTTP) as well. I'm sure you wouldn't want to double every e2e test for
both transports?

> 
> Again, let me point out that these tests should never replace the
> current tests but merely test at a different abstraction.

Yes, I agree this could prove useful. But in most scenarios I can think
of at the moment, more unit-like tests would be
+ simpler
+ easier to write and maintain
+ running much faster
+ succeed more often ;-)

But we're kind of talking very theoretical here (otherwise my answer
could have been shorter and there'd be somebody still listening down
here). Let's talk about a concrete example. For the moment, I'm -0 and
done issueing my little warning ;-)

  Bernd

Re: Vysper unit tests on running server

[Niklas Gustavsson <ni...@protocol7.com>]

On Sun, Jun 21, 2009 at 11:14 AM, Bernd Fondermann<bf...@brainlounge.de> wrote:
> For testing new functionality (mostly handlers), starting the server is
> not neccessary (and that's a big plus). Just testing the handler in
> isolation is usually sufficient.

I should have pointed out that I did not suggest to in any way replace
the current testing (nor step away from the design that allows
handlers and other classes to be tested in isloation), that should
still be done!

> At some point, we might need that integration tests, to make sure
> everything works properly together, though.

Yeah, that's kind of the experience we've had with FtpServer. The
integration tests has located tons of bugs, and allows us easily to
write tests for cases reported by users ("server does this when my
client does that").

> What do you intend to use this end-to-end testing for?

I would say that we should intend to test most, if not all, of the
client facing functionality this way (as well as with more
fine-grained test on each class).

> Integration tests for XMPP may prove difficult, because
> + XMPP is XML and the way elements are closed or allowance of arbitrary
> whitespaces can add for a lot of noise, elements order may be arbitrary etc.
> + stanzas may come out of order, too
> + extensions are manyfold
> + tracking session state is easier with our own testing facilities

I probably (surely) don't understand all the details here. But, my
idea would be to use Smack as the client which at least should isolate
us as much as we want from the protocol (but still allow us to create
arbitrary packages for testing odd cases). I also think this would
handle out of order stanzas and extensions. Not sure about session
state.

Again, let me point out that these tests should never replace the
current tests but merely test at a different abstraction.

/niklas

Re: Vysper unit tests on running server

[Bernd Fondermann <bf...@brainlounge.de>]

Niklas Gustavsson wrote:
> Hey
> 
> Over at FtpServer, we run a lot of unit tests but starting the server
> and then letting a client (in our case commons-net) interact with it,
> testing for correct outcomes. I'm thinking something similar would be
> a good idea for Vysper but fail to find any such tests (besides
> org.apache.vysper.smack.BasicClient but it looks more like something
> which is run interactively during development). What do you think?

For testing new functionality (mostly handlers), starting the server is
not neccessary (and that's a big plus). Just testing the handler in
isolation is usually sufficient.

At some point, we might need that integration tests, to make sure
everything works properly together, though.

What do you intend to use this end-to-end testing for?

Integration tests for XMPP may prove difficult, because
+ XMPP is XML and the way elements are closed or allowance of arbitrary
whitespaces can add for a lot of noise, elements order may be arbitrary etc.
+ stanzas may come out of order, too
+ extensions are manyfold
+ tracking session state is easier with our own testing facilities

 Bernd