You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Henri Yandell <fl...@gmail.com> on 2005/06/09 03:46:01 UTC
[csv] Creating a CSV component
Thought I'd summarise the CSV threads on the user list. Due to the
high level of user activity in those threads, I've mailed this to both
user and dev lists. Interested users should probably hop over to the
dev list at some point as I imagine future threads will consolidate
there.
Basic idea is for a parser library for csv files. There is a lot of
interest and various pieces of code have been offered:
Netcetera's csvparser.
- configurable delimiter (delimiter may appear in complex values)
- complex values (including newlines -> multiline values)
- unicode escapes
- empty line skipping support
- comment support
- hardcoded record separators (\n or \r\n)
OSJava's gj-csv (http://www.osjava.org/genjava/multiproject/gj-csv/)
- configurable field/block delimiters
- reader/writer, Map and 'DOM' apis
- BSD licenced.
- NOTE: My library, so I'm biased either for or against it.
Ostermiller's GPL'd csv parser has been around for ages, but is GPL.
Still, a good thing to be feature competitive with.
Brian McCallister has a new CSV library - http://kasparov.skife.org/csv/.
- Apache licenced.
There are alternative ways to do CSV parsing:
- JDBC API (http://csvjdbc.sourceforge.net/)
- using ANTLR
(http://supportweb.cs.bham.ac.uk/documentation/tutorials/docsystem/build/tutorials/antlr/antlr.html#ANTLR-Translation-Example)
- XML API (http://www.dpawson.co.uk/java/csv2xml.html)
Another question is where the code should go. [lang] and [io] have
been suggested, as has a [csv] component. I'll go as far as to say
that [csv] is the direction we should go and see if anybody disagrees
:)
There are important issues to remember:
* include a precise reference to a spec, if available, or to an
implementation (e.g. excel,
outlook, filemaker, ...)
* release early (ie) get the basics out).
* Excel and others can be weird, it might need special support.
Wish-list features:
* bridge with Jelly (whatever Paul meant here)
* configurable column selection.
* Hibernate / struts property driven CSV read configuration. (Here I
am talking about referencing third party xml elements as target
references.)
* xsl driven CSV conversions (CSV to XML, CSV to HTML, CSV to EDI, CSV
to *new format*)
*. CSVFilter as that for FileFilter -> column range, column width
range, row range
I'm prepared to help on a commons csv component. I obviously have the
itch/need for a basic csv library.
Opinions on where to start seem like the best direction.
* Netcetera
* OSJava
* Ostermiller (aka, people could ask for a licence change :) )
* Skife
* Start afresh
----
Looking at the Netcetera source, it looks to be nicely polished. Lots
of options to handle the CSV variations, fully javadoc'd and probably
100% test coverage (which is better than mine).
Src jar at:
ftp://ftp.netcetera.ch/pub/csvparser.jar
(not licenced for use currently, please don't use unless it becomes an
ASF codebase)
I'm interested in hearing any criticism of the Netcetera source.
API-wise, it's pretty much the way I think we all go with such a
problem, reader/writer-like. I added the concept of a CSV object to
mine as opposed to Netcetera's Object[][], and the CsvFieldReader is
for reading by column name and not index; but to be honest neither get
used by myself or at work.
If there's a good agreement to start with the Netcetera source, I need
to dig up the legalities on company contributions and guide Stefan and
Netcetera through them. I'm not sure if we'd want to add any features
before an initial release, but I think we'd definitely want to chug
along at adding more tests, if only to get fully inside the API.
Any thoughts?
Hen
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Henri Yandell <fl...@gmail.com>.
Sorry for not noticing this.
I have a fax from Stefan granting us the rights to use the Netcetera
code. I need to record this in the right place in svn, and talk to Jim
Jagielski to make sure we can proceed before I give him the faxes at
ApacheCon in a couple of weeks.
Sorry it's been so glacially slow, once we understand these things
better I think they'll go a lot quicker, just slow doing the first
few,
Hen
On 11/18/05, robert burrell donkin <ro...@blueyonder.co.uk> wrote:
> On Thu, 2005-11-17 at 09:36 -0600, Don Seiler wrote:
> > Stefan Rufer wrote:
> > > The discussion was continued on the commons-dev mailing list.
> > >
> > > Henri Yandell proposed to bring in the Netcetera CSV code via the Apache
> > > Incubator Project. I think that's a good idea and I will try to put some
> > > effort in to get the incubation process started.
> >
> > Is there any progress on this? I'll ask sf.net to remove the
> > csv4j.sf.net project if it seems Apache/Jakarta is carrying the ball now.
>
> AIUI we're processing the required paperwork now. hopefully it shouldn't
> be too much longer...
>
> - robert
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by robert burrell donkin <ro...@blueyonder.co.uk>.
On Thu, 2005-11-17 at 09:36 -0600, Don Seiler wrote:
> Stefan Rufer wrote:
> > The discussion was continued on the commons-dev mailing list.
> >
> > Henri Yandell proposed to bring in the Netcetera CSV code via the Apache
> > Incubator Project. I think that's a good idea and I will try to put some
> > effort in to get the incubation process started.
>
> Is there any progress on this? I'll ask sf.net to remove the
> csv4j.sf.net project if it seems Apache/Jakarta is carrying the ball now.
AIUI we're processing the required paperwork now. hopefully it shouldn't
be too much longer...
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Don Seiler <do...@seiler.us>.
Stefan Rufer wrote:
> The discussion was continued on the commons-dev mailing list.
>
> Henri Yandell proposed to bring in the Netcetera CSV code via the Apache
> Incubator Project. I think that's a good idea and I will try to put some
> effort in to get the incubation process started.
Is there any progress on this? I'll ask sf.net to remove the
csv4j.sf.net project if it seems Apache/Jakarta is carrying the ball now.
--
Don Seiler
don@seiler.us
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xFC87F041
Fingerprint: 0B56 50D5 E91E 4D4C 83B7 207C 76AC 5DA2 FC87 F041
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Stefan Rufer <st...@netcetera.ch>.
The discussion was continued on the commons-dev mailing list.
Henri Yandell proposed to bring in the Netcetera CSV code via the Apache
Incubator Project. I think that's a good idea and I will try to put some
effort in to get the incubation process started.
cu
Stefan
On Tue, 5 Jul 2005, Don Seiler wrote:
> From: Don Seiler <do...@seiler.us>
> To: Jakarta Commons Users List <co...@jakarta.apache.org>
> Subject: Re: [csv] Creating a CSV component
> Date: Tue, 5 Jul 2005 10:02:41 -0500
>
> On 10:28 Sun 19 Jun , robert burrell donkin wrote:
>> the discussions have moved onto dev but no consensus has been reached as
>> yet.
>>
>> On Tue, 2005-06-14 at 18:14 +0300, Emmanouil Batsis wrote:
>>> Just wanted to ask whether you folks have chosen what to use for this.
>
> Is there any more information on this? My situation has changed and if
> anyone else wants to work on something under the cvs4j.sf.net umbrella
> feel free to contact me and I'll transfer maintainership over to you.
> Personally I'd still prefer to see it under commons.
>
> --
> Don Seiler
> don@seiler.us
>
> Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xFC87F041
> Fingerprint: 0B56 50D5 E91E 4D4C 83B7 207C 76AC 5DA2 FC87 F041
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Don Seiler <do...@seiler.us>.
On 10:28 Sun 19 Jun , robert burrell donkin wrote:
> the discussions have moved onto dev but no consensus has been reached as
> yet.
>
> On Tue, 2005-06-14 at 18:14 +0300, Emmanouil Batsis wrote:
> > Just wanted to ask whether you folks have chosen what to use for this.
Is there any more information on this? My situation has changed and if
anyone else wants to work on something under the cvs4j.sf.net umbrella
feel free to contact me and I'll transfer maintainership over to you.
Personally I'd still prefer to see it under commons.
--
Don Seiler
don@seiler.us
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xFC87F041
Fingerprint: 0B56 50D5 E91E 4D4C 83B7 207C 76AC 5DA2 FC87 F041
Re: [csv] Creating a CSV component
Posted by robert burrell donkin <ro...@blueyonder.co.uk>.
the discussions have moved onto dev but no consensus has been reached as
yet.
- robert
On Tue, 2005-06-14 at 18:14 +0300, Emmanouil Batsis wrote:
> Just wanted to ask whether you folks have chosen what to use for this.
>
> Thanks,
>
> Manos
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Emmanouil Batsis <Em...@eurodyn.com>.
Just wanted to ask whether you folks have chosen what to use for this.
Thanks,
Manos
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Mark Diggory <md...@gmail.com>.
I would suggest that parsing files, especially data files is going to
have allot of overlap within commons. What about tab, fixed field etc.
This sounds more like its own tool in the long run. Think about the
implications for producing drivers for JDBC or parsers for JAXP etc.
thats based on a common parsing or reading api. Encompassing tools for
file type transformation etc. This is a very powerful subject, I suspect
it would be a project that could be dependent on codec, math and lang
for certain functionalities.
I'm listening in as well, our project is something that would enjoy the
advantages of such a parsing framework if it encompassed more than just
csv files.
-Mark Diggory
Federico Spinazzi wrote:
> Henri Yandell wrote:
>
>> Thought I'd summarise the CSV threads on the user list. Due to the
>> high level of user activity in those threads, I've mailed this to both
>> user and dev lists. Interested users should probably hop over to the
>> dev list at some point as I imagine future threads will consolidate
>> there.
>>
>> Basic idea is for a parser library for csv files. There is a lot of
>> interest and various pieces of code have been offered:
>>
>> Netcetera's csvparser.
>> OSJava's gj-csv (http://www.osjava.org/genjava/multiproject/gj-csv/)
>>
>> Ostermiller's GPL'd csv parser has been around for ages, but is GPL.
>> Still, a good thing to be feature competitive with.
>>
>> Brian McCallister has a new CSV library -
>> http://kasparov.skife.org/csv/.
>> There are alternative ways to do CSV parsing:
>>
>>
> You can also look at DbUnit csv classes. DbUnit is GPL, but as I've
> written them and I can re license as Apache 2.0 if I want (or not?).
> Federico
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Federico Spinazzi <f....@masterhouse.it>.
Henri Yandell wrote:
>Thought I'd summarise the CSV threads on the user list. Due to the
>high level of user activity in those threads, I've mailed this to both
>user and dev lists. Interested users should probably hop over to the
>dev list at some point as I imagine future threads will consolidate
>there.
>
>Basic idea is for a parser library for csv files. There is a lot of
>interest and various pieces of code have been offered:
>
>Netcetera's csvparser.
>
>OSJava's gj-csv (http://www.osjava.org/genjava/multiproject/gj-csv/)
>
>Ostermiller's GPL'd csv parser has been around for ages, but is GPL.
>Still, a good thing to be feature competitive with.
>
>Brian McCallister has a new CSV library - http://kasparov.skife.org/csv/.
>
>There are alternative ways to do CSV parsing:
>
>
You can also look at DbUnit csv classes. DbUnit is GPL, but as I've
written them and I can re license as Apache 2.0 if I want (or not?).
Federico
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Stefan Rufer <st...@netcetera.ch>.
Thanks for moving this, I am following the discussion with great interest.
On Wed, 8 Jun 2005, Henri Yandell wrote:
> If there's a good agreement to start with the Netcetera source, I need
> to dig up the legalities on company contributions and guide Stefan and
> Netcetera through them. I'm not sure if we'd want to add any features
I would like to mention that Netcetera has contributed the Websh framework
http://tcl.apache.org/websh/ and therefore has two committers there (Simon
Hefti, Ronnie Brunner). Maybe this simplifies upcoming legalities...?
cu
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Torsten Curdt <tc...@apache.org>.
Steven Caswell wrote:
> I'm also +1 to a separate [csv] component for the reasons already conveyed.
also think it's worth a separate component
cheers
--
Torsten
Re: [csv] Creating a CSV component
Posted by Steven Caswell <st...@gmail.com>.
I'm also +1 to a separate [csv] component for the reasons already conveyed.
On 6/9/05, Jeremias Maerki <de...@greenmail.ch> wrote:
>
> I must say that I agree with Henri that this merits its own component.
> IMO it doesn't fit into [lang] and would look strange within [io] or
> [codec]. CSV is a topic that comes up every now and then and if this
> could be solved once and for all then a separate component would be
> justified if only for it to be easily found which it wouldn't be if it
> were hidden in one of the existing packages.
>
> Too bad I don't have time to help. CSV stuff bit me more than a couple
> of times in the past. Thanks to Henri for putting some energy into it.
>
> On 09.06.2005 06:09:13 Kevin Gessner wrote:
> > On 6/8/05, Henri Yandell <fl...@gmail.com> wrote:
> > > <snip>
> > >
> > > Another question is where the code should go. [lang] and [io] have
> > > been suggested, as has a [csv] component. I'll go as far as to say
> > > that [csv] is the direction we should go and see if anybody disagrees
> > > :)
> >
> > Some sort of CVS utility sounds like a smashing idea. But I
> > respectfully disagree with your placement, Hen. I don't see why we
> > would need a whole new component for something that comfortably fits
> > into [io] or [lang].
>
>
> Jeremias Maerki
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
>
--
Steven Caswell
steven.caswell@gmail.com
Take back the web - http://www.mozilla.org
Re: [csv] Creating a CSV component
Posted by Jeremias Maerki <de...@greenmail.ch>.
I must say that I agree with Henri that this merits its own component.
IMO it doesn't fit into [lang] and would look strange within [io] or
[codec]. CSV is a topic that comes up every now and then and if this
could be solved once and for all then a separate component would be
justified if only for it to be easily found which it wouldn't be if it
were hidden in one of the existing packages.
Too bad I don't have time to help. CSV stuff bit me more than a couple
of times in the past. Thanks to Henri for putting some energy into it.
On 09.06.2005 06:09:13 Kevin Gessner wrote:
> On 6/8/05, Henri Yandell <fl...@gmail.com> wrote:
> > <snip>
> >
> > Another question is where the code should go. [lang] and [io] have
> > been suggested, as has a [csv] component. I'll go as far as to say
> > that [csv] is the direction we should go and see if anybody disagrees
> > :)
>
> Some sort of CVS utility sounds like a smashing idea. But I
> respectfully disagree with your placement, Hen. I don't see why we
> would need a whole new component for something that comfortably fits
> into [io] or [lang].
Jeremias Maerki
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [csv] Creating a CSV component
Posted by Kevin Gessner <wh...@gmail.com>.
On 6/8/05, Henri Yandell <fl...@gmail.com> wrote:
> <snip>
>
> Another question is where the code should go. [lang] and [io] have
> been suggested, as has a [csv] component. I'll go as far as to say
> that [csv] is the direction we should go and see if anybody disagrees
> :)
Some sort of CVS utility sounds like a smashing idea. But I
respectfully disagree with your placement, Hen. I don't see why we
would need a whole new component for something that comfortably fits
into [io] or [lang].
Cheers,
Kevin Gessner
> <more snip>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org