You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@commons.apache.org by "Ulbricht, Frank" <f....@qualitype.de> on 2014/11/28 09:13:23 UTC

[csv] Object Mapping Proposal

Hello there,

about 15 years ago I started to write a CSV library. Our company is using it for a long time now. Over the time a lot of interesting features were added. Now I have decided to replace it with the commons-csv. This API looks very good and it provides some low-level features our library is missing (e.g. multi-line records with escaping).

Nevertheless, we have a lot of high-level features for easily mapping between objects and the csv files. I am planning to migrate those to use the commons-csv for the I/O work. And I want to use this chance to redesign our APIs.

Now a thought crossed my mind, how about contributing this to the commons-csv? In order to get an idea want I am planning to do I have attached a sample project to this mail. It is just an idea, far from being perfect. It shall demonstrate how it may look like one day.

Sure, in the moment this sample uses a lot of pretty cool Java 8 stuff, but there are ways to make it available to earlier Java versions too.

Please have a look at the class "CSVObjectParserTest" it and tell me what you think.

Thank you,
Frank.

P.S. Looks like my previous mail was ignored because the subscription process was not yet finished. If not, please ignore the duplicate mail.

Re: [csv] Object Mapping Proposal

Posted by "Bruno P. Kinoshita" <br...@yahoo.com.br>.

Hi Matt!!
I had a look at flatfile, but to be honest I couldn't undestand how it works. Will check out the source code to play with it later.
Bruno

 
      From: Matt Benson <gu...@gmail.com>
 To: Bruno P. Kinoshita <br...@yahoo.com.br>; dev@commons.apache.org 
 Sent: Friday, November 28, 2014 4:57 PM
 Subject: Re: [csv] Object Mapping Proposal
   
Bruno, it sounds like you could have used the [flatfile] component from the
Commons sandbox. ;-)

Matt
On Nov 28, 2014 10:01 AM, "Bruno P. Kinoshita" <br...@yahoo.com.br>
wrote:

> Hi Frank,
> Is the project hosted in GitHub too?
> I think it sounds like a good and useful idea to include it in
> commons-csv. Last week I had to import some mainframe files using a Cobol
> copybook, with hundreds of fields, into Hadoop HDFS. I converted the file
> into a CSV, but some fields were multiline (or had binary content with \n,
> \r, etc).
>
> I had to hack the code and escape these values before using the data.
> Probably your code could have saved me some time :) I tried to use
> commons-csv and Python for pre-processing the csv after reading the Cobol,
> but with no luck.
>
> Thanks!Bruno
>
>
>      From: "Ulbricht, Frank" <f....@qualitype.de>
>  To: "dev@commons.apache.org" <de...@commons.apache.org>
>  Sent: Friday, November 28, 2014 6:13 AM
>  Subject: [csv] Object Mapping Proposal
>
>  <!--#yiv9591682121 _filtered #yiv9591682121 {font-family:"Cambria
> Math";panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv9591682121
> {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv9591682121
> #yiv9591682121 p.yiv9591682121MsoNormal, #yiv9591682121
> li.yiv9591682121MsoNormal, #yiv9591682121 div.yiv9591682121MsoNormal
> {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:"Times New
> Roman", serif;}#yiv9591682121 a:link, #yiv9591682121
> span.yiv9591682121MsoHyperlink
> {color:#0563C1;text-decoration:underline;}#yiv9591682121 a:visited,
> #yiv9591682121 span.yiv9591682121MsoHyperlinkFollowed
> {color:#954F72;text-decoration:underline;}#yiv9591682121
> span.yiv9591682121E-MailFormatvorlage17 {font-family:"Calibri",
> sans-serif;color:windowtext;}#yiv9591682121 .yiv9591682121MsoChpDefault
> {font-family:"Calibri", sans-serif;} _filtered #yiv9591682121
> {margin:70.85pt 70.85pt 2.0cm 70.85pt;}#yiv9591682121
> div.yiv9591682121WordSection1 {}-->Hello there,    about 15 years ago I


> started to write a CSV library. Our company is using it for a long time
> now. Over the time a lot of interesting features were added. Now I have
> decided to replace it with the commons-csv. This API looks very good and it
> provides some low-level features our library is missing (e.g. multi-line
> records with escaping).    Nevertheless, we have a lot of high-level
> features for easily mapping between objects and the csv files. I
> am planning to migrate those to use the commons-csv for the I/O work. And I
> want to use this chance to redesign our APIs.    Now a thought crossed my
> mind, how about contributing this to the commons-csv? In order to get an
> idea want I am planning to do I have attached a sample project to this
> mail. It is just an idea, far from being perfect. It shall demonstrate how
> it may look like one day.    Sure, in the moment this sample uses a lot of
> pretty cool Java 8 stuff, but there are ways to make it available to
> earlier Java versions too.    Please have a look at the class
> “CSVObjectParserTest” it and tell me what you think.    Thank you, Frank.
>  P.S. Looks like my previous mail was ignored because the subscription
> process was not yet finished. If not, please ignore the duplicate mail.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
>

Re: [csv] Object Mapping Proposal

Posted by Matt Benson <gu...@gmail.com>.

Bruno, it sounds like you could have used the [flatfile] component from the
Commons sandbox. ;-)

Matt
On Nov 28, 2014 10:01 AM, "Bruno P. Kinoshita" <br...@yahoo.com.br>
wrote:

> Hi Frank,
> Is the project hosted in GitHub too?
> I think it sounds like a good and useful idea to include it in
> commons-csv. Last week I had to import some mainframe files using a Cobol
> copybook, with hundreds of fields, into Hadoop HDFS. I converted the file
> into a CSV, but some fields were multiline (or had binary content with \n,
> \r, etc).
>
> I had to hack the code and escape these values before using the data.
> Probably your code could have saved me some time :) I tried to use
> commons-csv and Python for pre-processing the csv after reading the Cobol,
> but with no luck.
>
> Thanks!Bruno
>
>
>       From: "Ulbricht, Frank" <f....@qualitype.de>
>  To: "dev@commons.apache.org" <de...@commons.apache.org>
>  Sent: Friday, November 28, 2014 6:13 AM
>  Subject: [csv] Object Mapping Proposal
>
>  <!--#yiv9591682121 _filtered #yiv9591682121 {font-family:"Cambria
> Math";panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv9591682121
> {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv9591682121
> #yiv9591682121 p.yiv9591682121MsoNormal, #yiv9591682121
> li.yiv9591682121MsoNormal, #yiv9591682121 div.yiv9591682121MsoNormal
> {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:"Times New
> Roman", serif;}#yiv9591682121 a:link, #yiv9591682121
> span.yiv9591682121MsoHyperlink
> {color:#0563C1;text-decoration:underline;}#yiv9591682121 a:visited,
> #yiv9591682121 span.yiv9591682121MsoHyperlinkFollowed
> {color:#954F72;text-decoration:underline;}#yiv9591682121
> span.yiv9591682121E-MailFormatvorlage17 {font-family:"Calibri",
> sans-serif;color:windowtext;}#yiv9591682121 .yiv9591682121MsoChpDefault
> {font-family:"Calibri", sans-serif;} _filtered #yiv9591682121
> {margin:70.85pt 70.85pt 2.0cm 70.85pt;}#yiv9591682121
> div.yiv9591682121WordSection1 {}-->Hello there,    about 15 years ago I
> started to write a CSV library. Our company is using it for a long time
> now. Over the time a lot of interesting features were added. Now I have
> decided to replace it with the commons-csv. This API looks very good and it
> provides some low-level features our library is missing (e.g. multi-line
> records with escaping).    Nevertheless, we have a lot of high-level
> features for easily mapping between objects and the csv files. I
> am planning to migrate those to use the commons-csv for the I/O work. And I
> want to use this chance to redesign our APIs.    Now a thought crossed my
> mind, how about contributing this to the commons-csv? In order to get an
> idea want I am planning to do I have attached a sample project to this
> mail. It is just an idea, far from being perfect. It shall demonstrate how
> it may look like one day.    Sure, in the moment this sample uses a lot of
> pretty cool Java 8 stuff, but there are ways to make it available to
> earlier Java versions too.    Please have a look at the class
> “CSVObjectParserTest” it and tell me what you think.    Thank you, Frank.
>   P.S. Looks like my previous mail was ignored because the subscription
> process was not yet finished. If not, please ignore the duplicate mail.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
>

Re: [csv] Object Mapping Proposal

Posted by "Bruno P. Kinoshita" <br...@yahoo.com.br>.

Hi Frank,
Is the project hosted in GitHub too?
I think it sounds like a good and useful idea to include it in commons-csv. Last week I had to import some mainframe files using a Cobol copybook, with hundreds of fields, into Hadoop HDFS. I converted the file into a CSV, but some fields were multiline (or had binary content with \n, \r, etc).

I had to hack the code and escape these values before using the data. Probably your code could have saved me some time :) I tried to use commons-csv and Python for pre-processing the csv after reading the Cobol, but with no luck.

Thanks!Bruno

From: "Ulbricht, Frank" <f....@qualitype.de>
To: "dev@commons.apache.org" <de...@commons.apache.org>
Sent: Friday, November 28, 2014 6:13 AM
Subject: [csv] Object Mapping Proposal

Hello there, about 15 years ago I started to write a CSV library. Our company is using it for a long time now. Over the time a lot of interesting features were added. Now I have decided to replace it with the commons-csv. This API looks very good and it provides some low-level features our library is missing (e.g. multi-line records with escaping). Nevertheless, we have a lot of high-level features for easily mapping between objects and the csv files. I am planning to migrate those to use the commons-csv for the I/O work. And I want to use this chance to redesign our APIs. Now a thought crossed my mind, how about contributing this to the commons-csv? In order to get an idea want I am planning to do I have attached a sample project to this mail. It is just an idea, far from being perfect. It shall demonstrate how it may look like one day. Sure, in the moment this sample uses a lot of pretty cool Java 8 stuff, but there are ways to make it available to earlier Java versions too. Please have a look at the class “CSVObjectParserTest” it and tell me what you think. Thank you, Frank. P.S. Looks like my previous mail was ignored because the subscription process was not yet finished. If not, please ignore the duplicate mail.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org