You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "laisuchoa (via GitHub)" <gi...@apache.org> on 2024/08/14 16:47:03 UTC
[I] Query limit configuration not reflecting expected document retrieval count in large database [couchdb]
laisuchoa opened a new issue, #5176:
URL: https://github.com/apache/couchdb/issues/5176
# Description
We're working with a CouchDB database that holds about 272 million records. When we tried retrieving all documents using a curl command and writing to a file, we noticed that only 268435456 records were returned, despite the database having more entries.
The curl command in question:
`curl -k -X GET "https://$(cat credential_couchdb)@{host}/{db}/_all_docs?include_docs=true" > all_docs`
This led us to suspect a default query limit issue, which aligns with discussions in [this GitHub pull request](https://github.com/apache/couchdb/pull/1804).
To resolve this, we updated the config file to set both partition_query_limit and query_limit to 536870912.
`"query_server_config":{"partition_query_limit":"536870912","query_limit":"536870912"}`
Unfortunately, despite this configuration change, the output remained the same at 268435456 records. The _offset_ field from the output file correctly displays the total records as around 272 million, which confirms the database size but highlighted that the returned file count did not adjust post-configuration changes.
# Expected Behaviour
We expect the number of records in the output file to match the _offset_ field value, not to be stuck in this default value count.
# Additional Context
The CouchDB instance in question is integrated as a state database within a Hyperledger Fabric setup.
The database is updated once a day.
@lucasmation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29 [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed issue #5176: Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29
URL: https://github.com/apache/couchdb/issues/5176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29 [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed issue #5176: Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29
URL: https://github.com/apache/couchdb/issues/5176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29 [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on issue #5176:
URL: https://github.com/apache/couchdb/issues/5176#issuecomment-2297383786
https://github.com/apache/couchdb/blob/aca60ddf147811547b11cf5e8fd10bcfbe3c053e/src/couch_mrview/include/couch_mrview.hrl#L57
You may need to paginate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29 [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5176:
URL: https://github.com/apache/couchdb/issues/5176#issuecomment-2860296277
@laisuchoa thanks again for reaching out and reporting the issue. It should be fixed in the latest Apache CouchDB 3.5.0 version - both the default limit value is raised, and the setting should now take effect for `_all_docs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org