You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by magnetpest2k5 <gr...@gmail.com> on 2011/11/21 07:14:27 UTC
python couchdb data collection stops
I am using couchdb to store twitter data. I found that couchdb stops updating
its data base though I keep getting the twitter data. I basically store the
dictionary that contains twitter data by using the python couchdb save
method, db.save(twitter_dic) where db is the database instance. I find that
some times I get 3GB of data and couchdb stops storing, sometimes it stops
storing even when it reaches 0.6GB. I don't know what is the reason. If some
one have come across similar situation please help me out. Is there any
better method to store the data using python wrapper?
--
View this message in context: http://couchdb-development.1959287.n2.nabble.com/python-couchdb-data-collection-stops-tp7015488p7015488.html
Sent from the CouchDB Development mailing list archive at Nabble.com.
Re: python couchdb data collection stops
Posted by Noah Slater <ns...@tumbolia.org>.
Could you move this to the user list please?
http://couchdb.apache.org/community/lists.html
On Mon, Nov 21, 2011 at 6:01 PM, magnetpest2k5 <gr...@gmail.com> wrote:
> Thanks,
>
> I am not a database guy I dont know much about couchdb either. But from
> what
> I have found on the web is that each data base can go up to 7.5 GB with
> 512000, correct me if I am wrong. With that as threshold the couchdb stops
> some time at 0.6gb, some time it goes till 3.8 gb.
>
> As mentioned in my previous post I use couchdb to store twitter data. I am
> sure that the python program does not halt with any exception, the program
> keeps running I confirmed this by stopping the storage to couchdb database
> and instead storing the output in a CSV format. I found no problem in the
> python program. But when I started storing the python dictionary in couchdb
> I found that though the program keeps running the storage in couchdb has
> stopped.
>
> I dont know what is happening in log but by just seeing I can find these
> lines. I am storing the data on a database called "worldtweets4". Following
> are few lines of log where the problem might be as you can see the storage
> had stopped I dont get any more 'PUT" /databasename/ID.
>
> Mon, 21 Nov 2011 14:57:35 GMT] [info] [<0.17870.56>] 127.0.0.1 - - 'PUT'
> /worldtweets4/69084bd085994bc7b60ec8f586c69d8f 201
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27616.57>] 127.0.0.1 - - 'GET'
> /_all_dbs 200
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.23293.57>] 127.0.0.1 - - 'GET'
> /_session 200
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27615.57>] 127.0.0.1 - - 'GET'
> /
> 200
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27617.57>] 127.0.0.1 - - 'GET'
> /newdb/ 200
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27617.57>] 127.0.0.1 - - 'GET'
> /worldtweets4/ 200
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.23293.57>] 127.0.0.1 - - 'GET'
> /worldtweets3/ 200
>
> [Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27616.57>] 127.0.0.1 - - 'GET'
> /_users/ 200
>
> This same sequence gets repeated and then some where i get the following
>
> [Mon, 21 Nov 2011 15:59:02 GMT] [info] [<0.28139.57>] 127.0.0.1 - - 'GET'
>
> /worldtweets4/_all_docs?startkey=%22_design%2F%22&endkey=%22_design0%22&include_docs=true
> 200
>
> [Mon, 21 Nov 2011 15:59:02 GMT] [info] [<0.28140.57>] 127.0.0.1 - - 'GET'
> /_config/query_servers/ 200
>
> [Mon, 21 Nov 2011 15:59:02 GMT] [info] [<0.27620.57>] 127.0.0.1 - - 'GET'
> /worldtweets4/_all_docs?limit=11&descending=true 200
>
>
> Do you want me to fill this issue as ticket? in the link you have provided?
>
> Any help is greatly appreciated
>
> --
> View this message in context:
> http://couchdb-development.1959287.n2.nabble.com/python-couchdb-data-collection-stops-tp7015488p7017272.html
> Sent from the CouchDB Development mailing list archive at Nabble.com.
>
Re: python couchdb data collection stops
Posted by magnetpest2k5 <gr...@gmail.com>.
Thanks,
I am not a database guy I dont know much about couchdb either. But from what
I have found on the web is that each data base can go up to 7.5 GB with
512000, correct me if I am wrong. With that as threshold the couchdb stops
some time at 0.6gb, some time it goes till 3.8 gb.
As mentioned in my previous post I use couchdb to store twitter data. I am
sure that the python program does not halt with any exception, the program
keeps running I confirmed this by stopping the storage to couchdb database
and instead storing the output in a CSV format. I found no problem in the
python program. But when I started storing the python dictionary in couchdb
I found that though the program keeps running the storage in couchdb has
stopped.
I dont know what is happening in log but by just seeing I can find these
lines. I am storing the data on a database called "worldtweets4". Following
are few lines of log where the problem might be as you can see the storage
had stopped I dont get any more 'PUT" /databasename/ID.
Mon, 21 Nov 2011 14:57:35 GMT] [info] [<0.17870.56>] 127.0.0.1 - - 'PUT'
/worldtweets4/69084bd085994bc7b60ec8f586c69d8f 201
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27616.57>] 127.0.0.1 - - 'GET'
/_all_dbs 200
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.23293.57>] 127.0.0.1 - - 'GET'
/_session 200
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27615.57>] 127.0.0.1 - - 'GET' /
200
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27617.57>] 127.0.0.1 - - 'GET'
/newdb/ 200
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27617.57>] 127.0.0.1 - - 'GET'
/worldtweets4/ 200
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.23293.57>] 127.0.0.1 - - 'GET'
/worldtweets3/ 200
[Mon, 21 Nov 2011 15:50:29 GMT] [info] [<0.27616.57>] 127.0.0.1 - - 'GET'
/_users/ 200
This same sequence gets repeated and then some where i get the following
[Mon, 21 Nov 2011 15:59:02 GMT] [info] [<0.28139.57>] 127.0.0.1 - - 'GET'
/worldtweets4/_all_docs?startkey=%22_design%2F%22&endkey=%22_design0%22&include_docs=true
200
[Mon, 21 Nov 2011 15:59:02 GMT] [info] [<0.28140.57>] 127.0.0.1 - - 'GET'
/_config/query_servers/ 200
[Mon, 21 Nov 2011 15:59:02 GMT] [info] [<0.27620.57>] 127.0.0.1 - - 'GET'
/worldtweets4/_all_docs?limit=11&descending=true 200
Do you want me to fill this issue as ticket? in the link you have provided?
Any help is greatly appreciated
--
View this message in context: http://couchdb-development.1959287.n2.nabble.com/python-couchdb-data-collection-stops-tp7015488p7017272.html
Sent from the CouchDB Development mailing list archive at Nabble.com.
Re: python couchdb data collection stops
Posted by Alexander Shorin <kx...@gmail.com>.
On Mon, Nov 21, 2011 at 10:14 AM, magnetpest2k5 <gr...@gmail.com> wrote:
> I am using couchdb to store twitter data. I found that couchdb stops updating
> its data base though I keep getting the twitter data. I basically store the
> dictionary that contains twitter data by using the python couchdb save
> method, db.save(twitter_dic) where db is the database instance. I find that
> some times I get 3GB of data and couchdb stops storing, sometimes it stops
> storing even when it reaches 0.6GB. I don't know what is the reason. If some
> one have come across similar situation please help me out. Is there any
> better method to store the data using python wrapper?
>
Hi,
Few short thoughts:
- Get to know what error have you got: python one and what is in couch logs?
- Have you reached max document size?
- Try to store same data via curl - if this test passed, so there is
not CouchDB problem
- In last case you have to fill issue at ticket
http://code.google.com/p/couchdb-python/issues/list to let us to not
forget about this situation(;
P.S. IRL, imho, you have to keep documents smaller as they could be to
have lesser memory usage and faster documents processing by CouchDB
and it's clients.
--
,,,^..^,,,