[PATCH] CSV/TSVParser/DataStreamParser + tests + test fixes!

[Martin van den Bemt <ml...@mvdb.net>]

Hi everyone,

I am back ;)
I am not going to submit this into bugzilla (where the bug lives..), it
is too much files ;). We all know what it is about by now, if you missed
checkout, you are out of luck ;)

On request I made some working testcases, since I am almost doubling the
testcases for turbine-2, it is a very good effort I must say ;)

First of all : you cannot blindly use the attached patch.diff, since it
has some questionmarks in the beginning (not known to the current CVS
that is, but I left them there, to point out where they actually have to
live)
Here it goes :

- The TurbineDefault.properties is an minimum based property file, which
is enough for tests I am doing in CSV/TSVParser.java
- CSVParserTest.java and TSVParser.java are testcases for the
CVSParserTest.java and the TSVParser.java and are only runnable via
rttest.
- BaseTextCase.java is used by the only testcase currently in rttest
(XMLRPCtest or something, which is btw horribly slow in disconnecting
the socket..). This was not in cvs, so I am attaching it (it doesn't
actually do anything though..)
>From the diff :
- .cvsignore Added two cactus log files that are generated when running
the tests
- TSVParser.java and CSVParser.java : Hardly any code in it anymore, now
just calling super and setting the fieldSeperator. Also added my name :)
- DataStreamParser.java 
Changed a lot of code there. The formerly abstract initTokenizer, it has
changed from an abstract method to a non abstract, since all the
settings are the same for both parsers (or any other fieldseperator you
can think of).
You just have to call the initTokenizer() in super (see CSVParser.java)
and the DataStreamParser.setSeparator(char);, which will handle setting
the ordinarychar correctly.
Since I don't have a use case (at the top of the file this thing is used
from Velocity), it is hard to determine if I can remove the abstract
from the class and just let people call setSeparator where appropiate
and deprecate the CSV/TSVParsers. So feedback on this will be
appreciated, so I can change this behaviour. My intention is this :
I want to be able to use all kinds of seperated files from anywhere,
just by calling DataStreamParser and setting the fieldSeperator. 
I made it almost fool proof, so if you are a monkey, it will still work.
It now supports this :

- Empty fieldnames (say the field header is field1,,field3,,field5

It just makes up a fieldname where the fieldnames are empty.. It will
use the constant EMPTYFIELDNAME+the columnumber. If you actually type a
space there, you are too much a fool for me to handle, so that is not
covered ;)

- Empty field Values

The ValueParser will not show fields that don't have a value. So if you
think it should provide an empty string for whatever reason, let me know
I will put it in.

- Field names and field values

Of course, if you just supply all the values, it will still work.


Back to the diff... 
- cactus.properties  This was referencing the wrong port, so I changed
it to @maven.test.port@
- server.xml also adjusted the test.port to maven.test.port and the
appBase (added maven. in front).

How do I actually test this ?

- Install maven
- vi <your home directory>/build.properties
- add tdk.hom=<the location of catalina home> (the test build process
doesn't change catalina, so don't bother setting up a new catalina for
that).
- from the root type ant -f build-maven maven:uitest
- Wait patiently ;)
- You will see a lot of log4j errors, I couldn't fix that yet though.
- The tests (at least mine) should pass and you get a build successfull
;). 
If you think this is too much work for this commit you will find my test
output below (should be a fairly unique view, since I couldn't get the
rttests running at all from CVS) :

As you can see no errors ;)

Hope this is enough to get it committed...

Mvgr,
Martin van den Bemt


run-iutests:
    [junit] Running org.apache.turbine.services.cache.TurbineCacheTest
[RunServerTests] log4j:ERROR No appenders could be found for category
(org.apache.turbine.services.cache.TurbineCacheTest).
[RunServerTests] log4j:ERROR Please initialize the log4j system
properly.
     [java] log4j:ERROR No appenders could be found for category
(org.apache.turbine.services.cache.TurbineCacheTest).
     [java] log4j:ERROR Please initialize the log4j system properly.
     [java] Closing XML-RPC server socket.
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 215.545
sec
    [junit] Running org.apache.turbine.util.parser.BaseValueParserTest
[RunServerTests] log4j:ERROR No appenders could be found for category
(org.apache.turbine.util.parser.BaseValueParserTest).
[RunServerTests] log4j:ERROR Please initialize the log4j system
properly.
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 4.349
sec
    [junit] Running org.apache.turbine.util.CSVParserTest
[RunServerTests] log4j:ERROR No appenders could be found for category
(org.apache.turbine.util.CSVParserTest).
[RunServerTests] log4j:ERROR Please initialize the log4j system
properly.
     [java] testEmptyFieldValues
     [java] testNormalFieldValues
     [java] testEmptyFieldNames
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.439
sec
    [junit] Running org.apache.turbine.util.TSVParserTest
[RunServerTests] log4j:ERROR No appenders could be found for category
(org.apache.turbine.util.TSVParserTest).
[RunServerTests] log4j:ERROR Please initialize the log4j system
properly.
     [java] testEmptyFieldValues
     [java] testNormalFieldValues
     [java] testEmptyFieldNames
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.785
sec