You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by AnotherNetFellow <an...@gmail.com> on 2009/10/16 15:36:09 UTC
Importing from CSV or MySQL DUMP
Hi,
i'm testing CouchDB as an alternative to MySQL (using python rewriting my
application data model doesn't seem so difficult).
To have some bench i want to put in it something like 3 million entries and
see the get/put/delete latency. I have this data available in CSV format, or
in alternative to this i can import them in MySQL and then export everything
in SQL language.
Do you know if there is a simple way to import everything on couch db?
I think i can write a python script to read a csv file and put all entries
in couch, but hope something similar has already been written.
Thankyou very much
Giorgio
--
--
AnotherNetFellow
Email: anothernetfellow@gmail.com
Re: Importing from CSV or MySQL DUMP
Posted by Metin Akat <ak...@gmail.com>.
couchdb-python has quite decent documentation inside the source code
as doc strings.
you can read them like this:
import couchdb
help(couchdb)
And the API documentation isn't bad at all.
On Sat, Oct 17, 2009 at 3:11 PM, AnotherNetFellow
<an...@gmail.com> wrote:
> Hi Metin,
> I was using this one: http://couchdbkit.org/.
>
> I've downloaded couchdb-python, but i find it very difficult to be used.
> There isn't any tipe of documentation (but the API manual, that has some
> code examples, most of them broken).
>
> Any idea? Do you know where I can find some examples?
>
> Thankyou!
>
> 2009/10/16 Metin Akat <ak...@gmail.com>
>
>> Sure :)
>> If you don't know how to write couchdb-python models, here is a good
>> example:
>> http://bitbucket.org/bbangert/kai/src/c5a69c10aa0c/kai/model/human.py
>>
>> You can learn a lot by studying that application's source.
>>
>> Regards
>>
>>
>> On Fri, Oct 16, 2009 at 6:26 PM, AnotherNetFellow
>> <an...@gmail.com> wrote:
>> > Thankyou Metin.
>> > Just a question: are you using http://code.google.com/p/couchdb-python/?
>> >
>> > (i'm referring to the "import couchdb" line)
>> >
>> > Thankyou!
>> >
>> > 2009/10/16 Metin Akat <ak...@gmail.com>
>> >
>> >> Hi,
>> >> Writing such an import script is really simple. I doubt it that there
>> >> can be "something ready", because it would have to be very flexible in
>> >> order to do some real work.
>> >>
>> >> See the example I am attaching to this mail.
>> >> I use such scripts constantly.
>> >> Pay a special attention at the "bulk insert" part, otherwise you will
>> >> have serious performance issues.
>> >> The csv module I am importing here is not a part of the standard
>> >> library so you will have to install python-csv (or whatever is it
>> >> called in your distro).
>> >> Also read that module's documentation in order to choose the right csv
>> >> dialect for your needs.
>> >>
>> >> By no means this is the "perfect" way of importing csv documents, but
>> >> it does the job for me, I hope you'll find it useful too.
>> >>
>> >>
>> >> On Fri, Oct 16, 2009 at 4:36 PM, AnotherNetFellow
>> >> <an...@gmail.com> wrote:
>> >> > Hi,
>> >> > i'm testing CouchDB as an alternative to MySQL (using python rewriting
>> my
>> >> > application data model doesn't seem so difficult).
>> >> >
>> >> > To have some bench i want to put in it something like 3 million
>> entries
>> >> and
>> >> > see the get/put/delete latency. I have this data available in CSV
>> format,
>> >> or
>> >> > in alternative to this i can import them in MySQL and then export
>> >> everything
>> >> > in SQL language.
>> >> >
>> >> > Do you know if there is a simple way to import everything on couch db?
>> >> >
>> >> > I think i can write a python script to read a csv file and put all
>> >> entries
>> >> > in couch, but hope something similar has already been written.
>> >> >
>> >> > Thankyou very much
>> >> >
>> >> > Giorgio
>> >> >
>> >> > --
>> >> > --
>> >> > AnotherNetFellow
>> >> > Email: anothernetfellow@gmail.com
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > --
>> > AnotherNetFellow
>> > Email: anothernetfellow@gmail.com
>> >
>>
>
>
>
> --
> --
> AnotherNetFellow
> Email: anothernetfellow@gmail.com
>
Re: Importing from CSV or MySQL DUMP
Posted by Jim Kass <ji...@nhrevolution.com>.
Couchdbkit is probably better than couchdb-python. It has additional
modules for django as well.
As far as doing imports, you can use the csv module in python to read
a CSV into an array of dicts then you could load each dict as it's own
doc.
It shouldn't be much more effort to pull results directly from mysql,
though I imagine it'd be faster to process a single CSV.
-Jim
Sent from my iPhone
On Oct 17, 2009, at 7:45 AM, Jeremy Wall <jw...@google.com> wrote:
> Sent from my G1 google phone
>
> On Oct 17, 2009 7:12 AM, "AnotherNetFellow" <anothernetfellow@gmail.com
> >
> wrote:
>
> Hi Metin,
> I was using this one: http://couchdbkit.org/.
>
> I've downloaded couchdb-python, but i find it very difficult to be
> used.
> There isn't any tipe of documentation (but the API manual, that has
> some
> code examples, most of them broken).
>
> Any idea? Do you know where I can find some examples?
>
> Thankyou! 2009/10/16 Metin Akat <ak...@gmail.com> > Sure :) >
> If you
> don't know how to write...
> --
>
> -- AnotherNetFellow Email: anothernetfellow@gmail.com
Re: Importing from CSV or MySQL DUMP
Posted by Jeremy Wall <jw...@google.com>.
Sent from my G1 google phone
On Oct 17, 2009 7:12 AM, "AnotherNetFellow" <an...@gmail.com>
wrote:
Hi Metin,
I was using this one: http://couchdbkit.org/.
I've downloaded couchdb-python, but i find it very difficult to be used.
There isn't any tipe of documentation (but the API manual, that has some
code examples, most of them broken).
Any idea? Do you know where I can find some examples?
Thankyou! 2009/10/16 Metin Akat <ak...@gmail.com> > Sure :) > If you
don't know how to write...
--
-- AnotherNetFellow Email: anothernetfellow@gmail.com
Re: Importing from CSV or MySQL DUMP
Posted by AnotherNetFellow <an...@gmail.com>.
Hi Metin,
I was using this one: http://couchdbkit.org/.
I've downloaded couchdb-python, but i find it very difficult to be used.
There isn't any tipe of documentation (but the API manual, that has some
code examples, most of them broken).
Any idea? Do you know where I can find some examples?
Thankyou!
2009/10/16 Metin Akat <ak...@gmail.com>
> Sure :)
> If you don't know how to write couchdb-python models, here is a good
> example:
> http://bitbucket.org/bbangert/kai/src/c5a69c10aa0c/kai/model/human.py
>
> You can learn a lot by studying that application's source.
>
> Regards
>
>
> On Fri, Oct 16, 2009 at 6:26 PM, AnotherNetFellow
> <an...@gmail.com> wrote:
> > Thankyou Metin.
> > Just a question: are you using http://code.google.com/p/couchdb-python/?
> >
> > (i'm referring to the "import couchdb" line)
> >
> > Thankyou!
> >
> > 2009/10/16 Metin Akat <ak...@gmail.com>
> >
> >> Hi,
> >> Writing such an import script is really simple. I doubt it that there
> >> can be "something ready", because it would have to be very flexible in
> >> order to do some real work.
> >>
> >> See the example I am attaching to this mail.
> >> I use such scripts constantly.
> >> Pay a special attention at the "bulk insert" part, otherwise you will
> >> have serious performance issues.
> >> The csv module I am importing here is not a part of the standard
> >> library so you will have to install python-csv (or whatever is it
> >> called in your distro).
> >> Also read that module's documentation in order to choose the right csv
> >> dialect for your needs.
> >>
> >> By no means this is the "perfect" way of importing csv documents, but
> >> it does the job for me, I hope you'll find it useful too.
> >>
> >>
> >> On Fri, Oct 16, 2009 at 4:36 PM, AnotherNetFellow
> >> <an...@gmail.com> wrote:
> >> > Hi,
> >> > i'm testing CouchDB as an alternative to MySQL (using python rewriting
> my
> >> > application data model doesn't seem so difficult).
> >> >
> >> > To have some bench i want to put in it something like 3 million
> entries
> >> and
> >> > see the get/put/delete latency. I have this data available in CSV
> format,
> >> or
> >> > in alternative to this i can import them in MySQL and then export
> >> everything
> >> > in SQL language.
> >> >
> >> > Do you know if there is a simple way to import everything on couch db?
> >> >
> >> > I think i can write a python script to read a csv file and put all
> >> entries
> >> > in couch, but hope something similar has already been written.
> >> >
> >> > Thankyou very much
> >> >
> >> > Giorgio
> >> >
> >> > --
> >> > --
> >> > AnotherNetFellow
> >> > Email: anothernetfellow@gmail.com
> >> >
> >>
> >
> >
> >
> > --
> > --
> > AnotherNetFellow
> > Email: anothernetfellow@gmail.com
> >
>
--
--
AnotherNetFellow
Email: anothernetfellow@gmail.com
Re: Importing from CSV or MySQL DUMP
Posted by Metin Akat <ak...@gmail.com>.
Sure :)
If you don't know how to write couchdb-python models, here is a good example:
http://bitbucket.org/bbangert/kai/src/c5a69c10aa0c/kai/model/human.py
You can learn a lot by studying that application's source.
Regards
On Fri, Oct 16, 2009 at 6:26 PM, AnotherNetFellow
<an...@gmail.com> wrote:
> Thankyou Metin.
> Just a question: are you using http://code.google.com/p/couchdb-python/ ?
>
> (i'm referring to the "import couchdb" line)
>
> Thankyou!
>
> 2009/10/16 Metin Akat <ak...@gmail.com>
>
>> Hi,
>> Writing such an import script is really simple. I doubt it that there
>> can be "something ready", because it would have to be very flexible in
>> order to do some real work.
>>
>> See the example I am attaching to this mail.
>> I use such scripts constantly.
>> Pay a special attention at the "bulk insert" part, otherwise you will
>> have serious performance issues.
>> The csv module I am importing here is not a part of the standard
>> library so you will have to install python-csv (or whatever is it
>> called in your distro).
>> Also read that module's documentation in order to choose the right csv
>> dialect for your needs.
>>
>> By no means this is the "perfect" way of importing csv documents, but
>> it does the job for me, I hope you'll find it useful too.
>>
>>
>> On Fri, Oct 16, 2009 at 4:36 PM, AnotherNetFellow
>> <an...@gmail.com> wrote:
>> > Hi,
>> > i'm testing CouchDB as an alternative to MySQL (using python rewriting my
>> > application data model doesn't seem so difficult).
>> >
>> > To have some bench i want to put in it something like 3 million entries
>> and
>> > see the get/put/delete latency. I have this data available in CSV format,
>> or
>> > in alternative to this i can import them in MySQL and then export
>> everything
>> > in SQL language.
>> >
>> > Do you know if there is a simple way to import everything on couch db?
>> >
>> > I think i can write a python script to read a csv file and put all
>> entries
>> > in couch, but hope something similar has already been written.
>> >
>> > Thankyou very much
>> >
>> > Giorgio
>> >
>> > --
>> > --
>> > AnotherNetFellow
>> > Email: anothernetfellow@gmail.com
>> >
>>
>
>
>
> --
> --
> AnotherNetFellow
> Email: anothernetfellow@gmail.com
>
Re: Importing from CSV or MySQL DUMP
Posted by AnotherNetFellow <an...@gmail.com>.
Thankyou Metin.
Just a question: are you using http://code.google.com/p/couchdb-python/ ?
(i'm referring to the "import couchdb" line)
Thankyou!
2009/10/16 Metin Akat <ak...@gmail.com>
> Hi,
> Writing such an import script is really simple. I doubt it that there
> can be "something ready", because it would have to be very flexible in
> order to do some real work.
>
> See the example I am attaching to this mail.
> I use such scripts constantly.
> Pay a special attention at the "bulk insert" part, otherwise you will
> have serious performance issues.
> The csv module I am importing here is not a part of the standard
> library so you will have to install python-csv (or whatever is it
> called in your distro).
> Also read that module's documentation in order to choose the right csv
> dialect for your needs.
>
> By no means this is the "perfect" way of importing csv documents, but
> it does the job for me, I hope you'll find it useful too.
>
>
> On Fri, Oct 16, 2009 at 4:36 PM, AnotherNetFellow
> <an...@gmail.com> wrote:
> > Hi,
> > i'm testing CouchDB as an alternative to MySQL (using python rewriting my
> > application data model doesn't seem so difficult).
> >
> > To have some bench i want to put in it something like 3 million entries
> and
> > see the get/put/delete latency. I have this data available in CSV format,
> or
> > in alternative to this i can import them in MySQL and then export
> everything
> > in SQL language.
> >
> > Do you know if there is a simple way to import everything on couch db?
> >
> > I think i can write a python script to read a csv file and put all
> entries
> > in couch, but hope something similar has already been written.
> >
> > Thankyou very much
> >
> > Giorgio
> >
> > --
> > --
> > AnotherNetFellow
> > Email: anothernetfellow@gmail.com
> >
>
--
--
AnotherNetFellow
Email: anothernetfellow@gmail.com
Re: Importing from CSV or MySQL DUMP
Posted by Metin Akat <ak...@gmail.com>.
Hi,
Writing such an import script is really simple. I doubt it that there
can be "something ready", because it would have to be very flexible in
order to do some real work.
See the example I am attaching to this mail.
I use such scripts constantly.
Pay a special attention at the "bulk insert" part, otherwise you will
have serious performance issues.
The csv module I am importing here is not a part of the standard
library so you will have to install python-csv (or whatever is it
called in your distro).
Also read that module's documentation in order to choose the right csv
dialect for your needs.
By no means this is the "perfect" way of importing csv documents, but
it does the job for me, I hope you'll find it useful too.
On Fri, Oct 16, 2009 at 4:36 PM, AnotherNetFellow
<an...@gmail.com> wrote:
> Hi,
> i'm testing CouchDB as an alternative to MySQL (using python rewriting my
> application data model doesn't seem so difficult).
>
> To have some bench i want to put in it something like 3 million entries and
> see the get/put/delete latency. I have this data available in CSV format, or
> in alternative to this i can import them in MySQL and then export everything
> in SQL language.
>
> Do you know if there is a simple way to import everything on couch db?
>
> I think i can write a python script to read a csv file and put all entries
> in couch, but hope something similar has already been written.
>
> Thankyou very much
>
> Giorgio
>
> --
> --
> AnotherNetFellow
> Email: anothernetfellow@gmail.com
>