You are viewing a plain text version of this content. The canonical link for it is here.
Posted to announce@apache.org by Joan Touzet <wo...@apache.org> on 2017/08/07 18:09:26 UTC
[ANNOUNCE] Apache CouchDB 2.1.0 released
Dear community,
Apache CouchDB %VERSION% has been released and is available for
download.
CouchDB is a database that completely embraces the web. Store your data
with JSON documents. Access your documents with your web browser, via
HTTP. Query, combine, and transform your documents with JavaScript.
CouchDB works well with modern web and mobile apps. You can distribute
your data, efficiently using CouchDB’s incremental replication. CouchDB
supports master-master setups with automatic conflict detection.
CouchDB comes with a suite of features, such as on-the-fly document
transformation and real-time change notifications, that make web
development a breeze. It even comes with an easy to use web
administration console, served directly out of CouchDB! We care a lot
about distributed scaling. CouchDB is highly available and partition
tolerant, but is also eventually consistent. And we care a lot about
your data. CouchDB has a fault-tolerant storage engine that puts the
safety of your data first.
Download your copy here:
https://couchdb.apache.org/#download
Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
RHEL/CentOS are available.
CouchDB 2.1.0 is a feature release, and was originally published on
2017-08-07.
The community would like to thank all contributors for their part in
making this release, from the smallest bug report or patch to major
contributions in code, design, or marketing, we couldn’t have done it
without you!
The release notes follow. A formatted version of them can be viewed
at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
On behalf of the CouchDB PMC,
Joan Touzet
---
Version 2.1.0
The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
_scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
_scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
Other scheduling replicator improvements
Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
The 2.1.0 release also includes the following minor improvements:
COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
COUCHDB-2988: (mango) Allow query selector as changes and replication filter
COUCHDB-2992: Add additional support for document size
COUCHDB-3046: Improve reduce function overflow protection
COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
COUCHDB-3180: Add features list in the welcome message
COUCHDB-3203: Make auth handlers configurable (in ini files)
COUCHDB-3234: Track open shard timeouts with a counter instead of logging
COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
COUCHDB-3249: Add config to disable index all fields (text indexes)
COUCHDB-3251: Remove hot loop usage of filename:rootname/1
COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
COUCHDB-3298: Optimize writing btree nodes
COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
COUCHDB-3318: bypass couch_httpd_vhost if there are none
COUCHDB-3323: Idle dbs cause excessive overhead
COUCHDB-3324: Introduce couch_replicator_scheduler
COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
COUCHDB-3358: (mango) Use efficient set storage for field names
COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
#457: TeX/LaTeX/texinfo removed from default docs build chain
#469: (mango) Choose index based on fields match
#483: couchup database migration tool
#582: Add X-Frame-Options support to help protect against clickjacking
#593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
#624: Enable compaction daemon by default
#626: Allow enable node decom using string “true”
(mango) Configurable default limit, defaults to 25.
(mango) _design documents ignored when querying _all_docs
(mango) add $allMatch selector
Add local.d/default.d directories by default and document
Improved INSTALL.* text files
Fixed Issues
The 2.1.0 release includes fixes for the following issues:
COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
COUCHDB-2731: Authentication DB was not considered a system DB
COUCHDB-3010: (Superceded fix for replication exponential backoff)
COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
COUCHDB-3100: Fix documentation on require_valid_user
COUCHDB-3109: 500 when include_docs=true for linked documents
COUCHDB-3113: fabric:open_revs can return {ok, []}
COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
COUCHDB-3150: Update all shards with stale=update_after
COUCHDB-3158: Fix a crash when connection closes for _update
COUCHDB-3162: Default ssl settings cause a crash
COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
COUCHDB-3178: Fabric does not send message when filtering lots of documents
COUCHDB-3181: function_clause error when adding attachment to doc in _users db
COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
COUCHDB-3202: (mango) do not allow empty field names
COUCHDB-3220: Handle timeout in _revs_diff
COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
COUCHDB-3231: Allow fixing users’ documents (type and roles)
COUCHDB-3232: user context not passed down in fabric_view_all_docs
COUCHDB-3238: os_process_limit documentation wrong
COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
COUCHDB-3245: Make couchjs -S option take effect again
COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
COUCHDB-3255: Conflicts introduced by recreating docs with attachments
COUCHDB-3259: Don’t trap exits in couch_file
COUCHDB-3264: POST to _all_docs does not respect conflicts=true
COUCHDB-3269: view response can ‘hang’ with filter and limit specified
COUCHDB-3271: Replications crash with ‘kaboom’ exit
COUCHDB-3274: eof in couch_file can be incorrect after error
COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
COUCHDB-3289: handle error clause when calling fabric:open_revs
COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
COUCHDB-3364: Don’t crash compactor when compacting process fails.
COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
COUCHDB-3376: Fix mem3_shards under load
COUCHDB-3378: Fix mango full text detection
COUCHDB-3379: Fix couch_auth_cache reinitialization logic
COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
COUCHDB-3402: race condition in mem3 startup
#511: (mango) Return false for empty list
#595: Return 409 to PUT attachment with non-existent rev
#623: Ensure replicator _active_tasks entry reports recent pending changes value
#627: Pass UserCtx to fabric’s all_docs from mango query
#631: fix couchdb_os_proc_pool eunit timeouts
#644: Make couch_event_sup:stop/1 synchronous
#645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
#648: Fix couch_replicator_changes_reader:process_change
#649: Avoid a race when restarting an index updater
#667: Prevent a terrible race condition
#677: Make replication filter fetch error for _replicate return a 404
Fix CORS max_age configuration parameter via Access-Control-Max-Age
Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
Allow w parameter for attachments
Return “Bad Request” when count in /_uuids exceeds max
Fix crashes when replicator db is deleted
Skip internal replication if changes already replicated
Fix encoding issues on _update/../doc_id and PUT attachments
Upgrade Notes
The deprecated (and broken) OAuth 1.0 implementation has been removed.
If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by John Le Brasseur <jo...@gmail.com>.
Well done Joan.
On 7 August 2017 at 21:47, William Edney <be...@technicalpursuit.com>
wrote:
> Congrats to all, but especially to Joan whose hard work and undying
> persistence made this release happen!
>
> As a "basic" user of CouchDB, this release fixed a number of bugs for me
> and is going to allow me to use the Couch 2.X series for my clients and
> company. If there's ever any doubt to anyone on the committer team that
> your efforts make an impact, let me put that to rest - this release is
> making a huge difference for me!
>
> Cheers,
>
> - Bill
>
> On Mon, Aug 7, 2017 at 2:14 PM, Benjamin Anderson <b...@banjiewen.net> wrote:
>
> > > but a big and special thanks to Joan Touzet who made it her mission to
> > get this done and succeeded.
> >
> > Indeed. Thanks, Joan!
> >
> > --
> > b
> >
> > On Mon, Aug 7, 2017 at 11:53 AM, Robert Samuel Newson
> > <rn...@apache.org> wrote:
> > >
> > > Thanks to everyone for the amazing efforts exerted to get here but a
> big
> > and special thanks to Joan Touzet who made it her mission to get this
> done
> > and succeeded.
> > >
> > > Best,
> > > B.
> > >
> > >> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
> > >>
> > >> Dear community,
> > >>
> > >> Apache CouchDB 2.1.0 has been released and is available for
> > >> download.
> > >>
> > >> CouchDB is a database that completely embraces the web. Store your
> data
> > >> with JSON documents. Access your documents with your web browser, via
> > >> HTTP. Query, combine, and transform your documents with JavaScript.
> > >> CouchDB works well with modern web and mobile apps. You can distribute
> > >> your data, efficiently using CouchDB’s incremental replication.
> CouchDB
> > >> supports master-master setups with automatic conflict detection.
> > >>
> > >> CouchDB comes with a suite of features, such as on-the-fly document
> > >> transformation and real-time change notifications, that make web
> > >> development a breeze. It even comes with an easy to use web
> > >> administration console, served directly out of CouchDB! We care a lot
> > >> about distributed scaling. CouchDB is highly available and partition
> > >> tolerant, but is also eventually consistent. And we care a lot about
> > >> your data. CouchDB has a fault-tolerant storage engine that puts the
> > >> safety of your data first.
> > >>
> > >> Download your copy here:
> > >>
> > >> https://couchdb.apache.org/#download
> > >>
> > >> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> > >> RHEL/CentOS are available.
> > >>
> > >> CouchDB 2.1.0 is a feature release, and was originally published on
> > >> 2017-08-07.
> > >>
> > >> The community would like to thank all contributors for their part in
> > >> making this release, from the smallest bug report or patch to major
> > >> contributions in code, design, or marketing, we couldn’t have done it
> > >> without you!
> > >>
> > >> The release notes follow. A formatted version of them can be viewed
> > >> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
> > >>
> > >> On behalf of the CouchDB PMC,
> > >> Joan Touzet
> > >>
> > >> ---
> > >>
> > >> Version 2.1.0
> > >>
> > >> The Mango _find endpoint supports a new combination operator,
> > $allMatch, which matches and returns all documents that contain an array
> > field with all its elements matching all the specified query criteria.
> > >>
> > >> New scheduling replicator. The core of the new replicator is a
> > scheduler which allows running a large number of replication jobs by
> > switching between them, stopping some and starting others periodically.
> > Jobs which fail are backed off exponentially. There is also an improved
> > inspection and querying API: _scheduler/jobs and _scheduler/docs:
> > >> _scheduler/jobs : This endpoint shows active replication jobs.
> > These are jobs managed by the scheduler. Some of them might be running,
> > some might be waiting to run, or backed off (penalized) because they
> > crashed too many times. Semantically this is somewhat equivalent to
> > _active_tasks but focuses only on replications. Jobs which have completed
> > or which were never created because of malformed replication documents
> will
> > not be shown here as they are not managed by the scheduler. _replicate
> > replications, started form _replicate endpoint not from a document in a
> > _replicator db, will also show up here.
> > >> _scheduler/docs : This endpoint is an improvement on having to
> > go back and read replication documents to query their state. It
> represents
> > the state of all the replications started from documents in _replicator
> db.
> > Unlike _scheduler/jobs it will also show jobs which have failed or have
> > completed.
> > >>
> > >> By default, scheduling replicator will not update documents with
> > transient states like triggered or error anymore, instead _scheduler/docs
> > API should be used to query replication document states.
> > >>
> > >> Other scheduling replicator improvements
> > >>
> > >> Network resource usage and performance was improved by
> > implementing a shared connection pool. This should help in cases of a
> large
> > number of connections to the same sources or target. Previously
> connection
> > pools were shared only withing a single replication job.
> > >> Improved request rate limit handling. Replicator requests will
> > auto-discover rate limit capacity on targets and sources based on a
> proven
> > Additive Increase / Multiplicative Decrease feedback control algorithm.
> > >> Improved performance by having exponential backoff for all
> > replication jobs failures. Previously there were some scenarios were
> > failure led to continuous repeated retries, consuming CPU and disk
> > resources in the process.
> > >> Improved recovery from long but temporary network failure.
> > Currently if replications jobs fail to start 10 times in a row, they will
> > not be retried anymore. This is sometimes desirable, but in some cases,
> for
> > example, after a sustained DNS failure which eventually recovers,
> > replications reach their retry limit, stop retrying and never recover.
> > Previously it required user intervention to continue. Scheduling
> replicator
> > will never give up retrying a valid scheduled replication job and so it
> > should recover automatically.
> > >> Better handling of filtered replications. Failing user filter
> > code fetches from the source will not block replicator manager and stall
> > other replications. Failing filter fetches will also be backed off
> > exponentially. Another improvement is when filter code changes on the
> > source, a running replication will detect that and restart itself with a
> > new replication ID automatically.
> > >>
> > >> The 2.1.0 release also includes the following minor improvements:
> > >>
> > >> COUCHDB-1946: Hibernate couch_stream after each write (up to 70%
> > reduction in memory usage during replication of DBs with large
> attachments)
> > >> COUCHDB-2964: Investigate switching replicator manager change feeds
> > to using “normal” instead of “longpoll”
> > >> COUCHDB-2988: (mango) Allow query selector as changes and
> > replication filter
> > >> COUCHDB-2992: Add additional support for document size
> > >> COUCHDB-3046: Improve reduce function overflow protection
> > >> COUCHDB-3061: Use vectored reads to search for buried headers in
> > .couch files. “On a modern linux system with SSD, we see improvements up
> to
> > 15x.”
> > >> COUCHDB-3063: “stale=ok” option replaced with new “stable” and
> > “update” options.
> > >> COUCHDB-3180: Add features list in the welcome message
> > >> COUCHDB-3203: Make auth handlers configurable (in ini files)
> > >> COUCHDB-3234: Track open shard timeouts with a counter instead of
> > logging
> > >> COUCHDB-3242: Make get view group info timeout in couch_indexer
> > configurable
> > >> COUCHDB-3249: Add config to disable index all fields (text indexes)
> > >> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> > >> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and
> > binary memory usage
> > >> COUCHDB-3298: Optimize writing btree nodes
> > >> COUCHDB-3302: (Improve) Attachment replication over low bandwidth
> > network connections
> > >> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per
> > db open
> > >> COUCHDB-3318: bypass couch_httpd_vhost if there are none
> > >> COUCHDB-3323: Idle dbs cause excessive overhead
> > >> COUCHDB-3324: Introduce couch_replicator_scheduler
> > >> COUCHDB-3337: End-point _local_docs doesn’t conform to query params
> > of _all_docs
> > >> COUCHDB-3358: (mango) Use efficient set storage for field names
> > >> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit
> > configurable
> > >> #457: TeX/LaTeX/texinfo removed from default docs build chain
> > >> #469: (mango) Choose index based on fields match
> > >> #483: couchup database migration tool
> > >> #582: Add X-Frame-Options support to help protect against
> > clickjacking
> > >> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single
> > nodes
> > >> #624: Enable compaction daemon by default
> > >> #626: Allow enable node decom using string “true”
> > >> (mango) Configurable default limit, defaults to 25.
> > >> (mango) _design documents ignored when querying _all_docs
> > >> (mango) add $allMatch selector
> > >> Add local.d/default.d directories by default and document
> > >> Improved INSTALL.* text files
> > >>
> > >> Fixed Issues
> > >>
> > >> The 2.1.0 release includes fixes for the following issues:
> > >>
> > >> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom
> > headers are specified in response of _update handler (missed in 2.0
> merge)
> > >> COUCHDB-2731: Authentication DB was not considered a system DB
> > >> COUCHDB-3010: (Superceded fix for replication exponential backoff)
> > >> COUCHDB-3090: Error when handling empty “Access-Control-Request-
> Headers”
> > header
> > >> COUCHDB-3100: Fix documentation on require_valid_user
> > >> COUCHDB-3109: 500 when include_docs=true for linked documents
> > >> COUCHDB-3113: fabric:open_revs can return {ok, []}
> > >> COUCHDB-3149: Exception written to the log if db deleted while
> there
> > is a change feed running
> > >> COUCHDB-3150: Update all shards with stale=update_after
> > >> COUCHDB-3158: Fix a crash when connection closes for _update
> > >> COUCHDB-3162: Default ssl settings cause a crash
> > >> COUCHDB-3164: Request fails when using _changes?feed=eventsource&
> > heartbeat=30000
> > >> COUCHDB-3168: Replicator doesn’t handle well writing documents to a
> > target db which has a small max_document_size
> > >> COUCHDB-3173: Views return corrupt data for text fields containing
> > non-BMP characters
> > >> COUCHDB-3174: max_document_size setting can by bypassed by issuing
> > multipart/related requests
> > >> COUCHDB-3178: Fabric does not send message when filtering lots of
> > documents
> > >> COUCHDB-3181: function_clause error when adding attachment to doc
> in
> > _users db
> > >> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle
> > errors in spawned process
> > >> COUCHDB-3193: fabric:open_revs returns multiple results when one of
> > the shards has stem_interactive_updates=false
> > >> COUCHDB-3199: Replicator VDU function doesn’t acount for an already
> > malformed document in replicator db
> > >> COUCHDB-3202: (mango) do not allow empty field names
> > >> COUCHDB-3220: Handle timeout in _revs_diff
> > >> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key
> > during document creation
> > >> COUCHDB-3231: Allow fixing users’ documents (type and roles)
> > >> COUCHDB-3232: user context not passed down in fabric_view_all_docs
> > >> COUCHDB-3238: os_process_limit documentation wrong
> > >> COUCHDB-3241: race condition in couch_server if delete msg for a db
> > is received before open_result msg
> > >> COUCHDB-3245: Make couchjs -S option take effect again
> > >> COUCHDB-3252: Include main-coffee.js in release artifact (broken
> > CoffeeScript view server)
> > >> COUCHDB-3255: Conflicts introduced by recreating docs with
> > attachments
> > >> COUCHDB-3259: Don’t trap exits in couch_file
> > >> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> > >> COUCHDB-3269: view response can ‘hang’ with filter and limit
> > specified
> > >> COUCHDB-3271: Replications crash with ‘kaboom’ exit
> > >> COUCHDB-3274: eof in couch_file can be incorrect after error
> > >> COUCHDB-3277: Replication manager crashes when it finds _replicator
> > db shards which are not part of a mem3 db
> > >> COUCHDB-3286: Validation function throwing unexpected json crashes
> > with function_clause
> > >> COUCHDB-3289: handle error clause when calling fabric:open_revs
> > >> COUCHDB-3291: Excessively long document IDs prevent replicator from
> > making progress
> > >> COUCHDB-3293: Allow limiting length of document ID (for CouchDB
> > proper)
> > >> COUCHDB-3305: (mango) don’t crash with invalid input to built in
> > reducer function
> > >> COUCHDB-3362: DELETE attachment on non-existing document creates
> the
> > document, rather than returning 404
> > >> COUCHDB-3364: Don’t crash compactor when compacting process fails.
> > >> COUCHDB-3367: Require server admin user for db/_compact and
> > db_view_cleanup endpoints
> > >> COUCHDB-3376: Fix mem3_shards under load
> > >> COUCHDB-3378: Fix mango full text detection
> > >> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> > >> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc
> > updated
> > >> COUCHDB-3402: race condition in mem3 startup
> > >> #511: (mango) Return false for empty list
> > >> #595: Return 409 to PUT attachment with non-existent rev
> > >> #623: Ensure replicator _active_tasks entry reports recent pending
> > changes value
> > >> #627: Pass UserCtx to fabric’s all_docs from mango query
> > >> #631: fix couchdb_os_proc_pool eunit timeouts
> > >> #644: Make couch_event_sup:stop/1 synchronous
> > >> #645: Pass db open options to fabric_view_map for _view and _list
> > queries on _users DB
> > >> #648: Fix couch_replicator_changes_reader:process_change
> > >> #649: Avoid a race when restarting an index updater
> > >> #667: Prevent a terrible race condition
> > >> #677: Make replication filter fetch error for _replicate return a
> 404
> > >> Fix CORS max_age configuration parameter via Access-Control-Max-Age
> > >> Chunk missing revisions before attempting to save on target
> > (improves replication for very conflicted, very deep revision tree
> > documents)
> > >> Allow w parameter for attachments
> > >> Return “Bad Request” when count in /_uuids exceeds max
> > >> Fix crashes when replicator db is deleted
> > >> Skip internal replication if changes already replicated
> > >> Fix encoding issues on _update/../doc_id and PUT attachments
> > >>
> > >> Upgrade Notes
> > >>
> > >> The deprecated (and broken) OAuth 1.0 implementation has been
> > removed.
> > >> If user code reads or manipulates replicator document states,
> > consider using the [replicator] update_docs = true compatibility
> parameter.
> > In that case replicator will continue updating documents with transient
> > replication states. However, that will incure a performance cost.
> Consider
> > instead switching using _scheduler/docs HTTP endpoint.
> > >> The stale parameter for views and _find has been deprecated in
> > favour of two new parameters: stable and update. The old stale=ok
> behaviour
> > is equivalent to stable=true&update=false, and the old stale=update_after
> > behaviour is equivalent to stable=true&update=lazy. The deprecated stale
> > parameter will be removed in CouchDB 3.0.
> > >> The [couchdb] max_document_size configuration parameter was renamed
> > to [httpd] max_http_request_size to reflect its actual implementation. It
> > has been enhanced by a new [couchdb] single_max_doc_size setting. Both
> are
> > documented in the shipped default.ini file.
> > >
> >
>
John Le Brasseur
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by William Edney <be...@technicalpursuit.com>.
Congrats to all, but especially to Joan whose hard work and undying
persistence made this release happen!
As a "basic" user of CouchDB, this release fixed a number of bugs for me
and is going to allow me to use the Couch 2.X series for my clients and
company. If there's ever any doubt to anyone on the committer team that
your efforts make an impact, let me put that to rest - this release is
making a huge difference for me!
Cheers,
- Bill
On Mon, Aug 7, 2017 at 2:14 PM, Benjamin Anderson <b...@banjiewen.net> wrote:
> > but a big and special thanks to Joan Touzet who made it her mission to
> get this done and succeeded.
>
> Indeed. Thanks, Joan!
>
> --
> b
>
> On Mon, Aug 7, 2017 at 11:53 AM, Robert Samuel Newson
> <rn...@apache.org> wrote:
> >
> > Thanks to everyone for the amazing efforts exerted to get here but a big
> and special thanks to Joan Touzet who made it her mission to get this done
> and succeeded.
> >
> > Best,
> > B.
> >
> >> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
> >>
> >> Dear community,
> >>
> >> Apache CouchDB 2.1.0 has been released and is available for
> >> download.
> >>
> >> CouchDB is a database that completely embraces the web. Store your data
> >> with JSON documents. Access your documents with your web browser, via
> >> HTTP. Query, combine, and transform your documents with JavaScript.
> >> CouchDB works well with modern web and mobile apps. You can distribute
> >> your data, efficiently using CouchDB’s incremental replication. CouchDB
> >> supports master-master setups with automatic conflict detection.
> >>
> >> CouchDB comes with a suite of features, such as on-the-fly document
> >> transformation and real-time change notifications, that make web
> >> development a breeze. It even comes with an easy to use web
> >> administration console, served directly out of CouchDB! We care a lot
> >> about distributed scaling. CouchDB is highly available and partition
> >> tolerant, but is also eventually consistent. And we care a lot about
> >> your data. CouchDB has a fault-tolerant storage engine that puts the
> >> safety of your data first.
> >>
> >> Download your copy here:
> >>
> >> https://couchdb.apache.org/#download
> >>
> >> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> >> RHEL/CentOS are available.
> >>
> >> CouchDB 2.1.0 is a feature release, and was originally published on
> >> 2017-08-07.
> >>
> >> The community would like to thank all contributors for their part in
> >> making this release, from the smallest bug report or patch to major
> >> contributions in code, design, or marketing, we couldn’t have done it
> >> without you!
> >>
> >> The release notes follow. A formatted version of them can be viewed
> >> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
> >>
> >> On behalf of the CouchDB PMC,
> >> Joan Touzet
> >>
> >> ---
> >>
> >> Version 2.1.0
> >>
> >> The Mango _find endpoint supports a new combination operator,
> $allMatch, which matches and returns all documents that contain an array
> field with all its elements matching all the specified query criteria.
> >>
> >> New scheduling replicator. The core of the new replicator is a
> scheduler which allows running a large number of replication jobs by
> switching between them, stopping some and starting others periodically.
> Jobs which fail are backed off exponentially. There is also an improved
> inspection and querying API: _scheduler/jobs and _scheduler/docs:
> >> _scheduler/jobs : This endpoint shows active replication jobs.
> These are jobs managed by the scheduler. Some of them might be running,
> some might be waiting to run, or backed off (penalized) because they
> crashed too many times. Semantically this is somewhat equivalent to
> _active_tasks but focuses only on replications. Jobs which have completed
> or which were never created because of malformed replication documents will
> not be shown here as they are not managed by the scheduler. _replicate
> replications, started form _replicate endpoint not from a document in a
> _replicator db, will also show up here.
> >> _scheduler/docs : This endpoint is an improvement on having to
> go back and read replication documents to query their state. It represents
> the state of all the replications started from documents in _replicator db.
> Unlike _scheduler/jobs it will also show jobs which have failed or have
> completed.
> >>
> >> By default, scheduling replicator will not update documents with
> transient states like triggered or error anymore, instead _scheduler/docs
> API should be used to query replication document states.
> >>
> >> Other scheduling replicator improvements
> >>
> >> Network resource usage and performance was improved by
> implementing a shared connection pool. This should help in cases of a large
> number of connections to the same sources or target. Previously connection
> pools were shared only withing a single replication job.
> >> Improved request rate limit handling. Replicator requests will
> auto-discover rate limit capacity on targets and sources based on a proven
> Additive Increase / Multiplicative Decrease feedback control algorithm.
> >> Improved performance by having exponential backoff for all
> replication jobs failures. Previously there were some scenarios were
> failure led to continuous repeated retries, consuming CPU and disk
> resources in the process.
> >> Improved recovery from long but temporary network failure.
> Currently if replications jobs fail to start 10 times in a row, they will
> not be retried anymore. This is sometimes desirable, but in some cases, for
> example, after a sustained DNS failure which eventually recovers,
> replications reach their retry limit, stop retrying and never recover.
> Previously it required user intervention to continue. Scheduling replicator
> will never give up retrying a valid scheduled replication job and so it
> should recover automatically.
> >> Better handling of filtered replications. Failing user filter
> code fetches from the source will not block replicator manager and stall
> other replications. Failing filter fetches will also be backed off
> exponentially. Another improvement is when filter code changes on the
> source, a running replication will detect that and restart itself with a
> new replication ID automatically.
> >>
> >> The 2.1.0 release also includes the following minor improvements:
> >>
> >> COUCHDB-1946: Hibernate couch_stream after each write (up to 70%
> reduction in memory usage during replication of DBs with large attachments)
> >> COUCHDB-2964: Investigate switching replicator manager change feeds
> to using “normal” instead of “longpoll”
> >> COUCHDB-2988: (mango) Allow query selector as changes and
> replication filter
> >> COUCHDB-2992: Add additional support for document size
> >> COUCHDB-3046: Improve reduce function overflow protection
> >> COUCHDB-3061: Use vectored reads to search for buried headers in
> .couch files. “On a modern linux system with SSD, we see improvements up to
> 15x.”
> >> COUCHDB-3063: “stale=ok” option replaced with new “stable” and
> “update” options.
> >> COUCHDB-3180: Add features list in the welcome message
> >> COUCHDB-3203: Make auth handlers configurable (in ini files)
> >> COUCHDB-3234: Track open shard timeouts with a counter instead of
> logging
> >> COUCHDB-3242: Make get view group info timeout in couch_indexer
> configurable
> >> COUCHDB-3249: Add config to disable index all fields (text indexes)
> >> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> >> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and
> binary memory usage
> >> COUCHDB-3298: Optimize writing btree nodes
> >> COUCHDB-3302: (Improve) Attachment replication over low bandwidth
> network connections
> >> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per
> db open
> >> COUCHDB-3318: bypass couch_httpd_vhost if there are none
> >> COUCHDB-3323: Idle dbs cause excessive overhead
> >> COUCHDB-3324: Introduce couch_replicator_scheduler
> >> COUCHDB-3337: End-point _local_docs doesn’t conform to query params
> of _all_docs
> >> COUCHDB-3358: (mango) Use efficient set storage for field names
> >> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit
> configurable
> >> #457: TeX/LaTeX/texinfo removed from default docs build chain
> >> #469: (mango) Choose index based on fields match
> >> #483: couchup database migration tool
> >> #582: Add X-Frame-Options support to help protect against
> clickjacking
> >> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single
> nodes
> >> #624: Enable compaction daemon by default
> >> #626: Allow enable node decom using string “true”
> >> (mango) Configurable default limit, defaults to 25.
> >> (mango) _design documents ignored when querying _all_docs
> >> (mango) add $allMatch selector
> >> Add local.d/default.d directories by default and document
> >> Improved INSTALL.* text files
> >>
> >> Fixed Issues
> >>
> >> The 2.1.0 release includes fixes for the following issues:
> >>
> >> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom
> headers are specified in response of _update handler (missed in 2.0 merge)
> >> COUCHDB-2731: Authentication DB was not considered a system DB
> >> COUCHDB-3010: (Superceded fix for replication exponential backoff)
> >> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers”
> header
> >> COUCHDB-3100: Fix documentation on require_valid_user
> >> COUCHDB-3109: 500 when include_docs=true for linked documents
> >> COUCHDB-3113: fabric:open_revs can return {ok, []}
> >> COUCHDB-3149: Exception written to the log if db deleted while there
> is a change feed running
> >> COUCHDB-3150: Update all shards with stale=update_after
> >> COUCHDB-3158: Fix a crash when connection closes for _update
> >> COUCHDB-3162: Default ssl settings cause a crash
> >> COUCHDB-3164: Request fails when using _changes?feed=eventsource&
> heartbeat=30000
> >> COUCHDB-3168: Replicator doesn’t handle well writing documents to a
> target db which has a small max_document_size
> >> COUCHDB-3173: Views return corrupt data for text fields containing
> non-BMP characters
> >> COUCHDB-3174: max_document_size setting can by bypassed by issuing
> multipart/related requests
> >> COUCHDB-3178: Fabric does not send message when filtering lots of
> documents
> >> COUCHDB-3181: function_clause error when adding attachment to doc in
> _users db
> >> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle
> errors in spawned process
> >> COUCHDB-3193: fabric:open_revs returns multiple results when one of
> the shards has stem_interactive_updates=false
> >> COUCHDB-3199: Replicator VDU function doesn’t acount for an already
> malformed document in replicator db
> >> COUCHDB-3202: (mango) do not allow empty field names
> >> COUCHDB-3220: Handle timeout in _revs_diff
> >> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key
> during document creation
> >> COUCHDB-3231: Allow fixing users’ documents (type and roles)
> >> COUCHDB-3232: user context not passed down in fabric_view_all_docs
> >> COUCHDB-3238: os_process_limit documentation wrong
> >> COUCHDB-3241: race condition in couch_server if delete msg for a db
> is received before open_result msg
> >> COUCHDB-3245: Make couchjs -S option take effect again
> >> COUCHDB-3252: Include main-coffee.js in release artifact (broken
> CoffeeScript view server)
> >> COUCHDB-3255: Conflicts introduced by recreating docs with
> attachments
> >> COUCHDB-3259: Don’t trap exits in couch_file
> >> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> >> COUCHDB-3269: view response can ‘hang’ with filter and limit
> specified
> >> COUCHDB-3271: Replications crash with ‘kaboom’ exit
> >> COUCHDB-3274: eof in couch_file can be incorrect after error
> >> COUCHDB-3277: Replication manager crashes when it finds _replicator
> db shards which are not part of a mem3 db
> >> COUCHDB-3286: Validation function throwing unexpected json crashes
> with function_clause
> >> COUCHDB-3289: handle error clause when calling fabric:open_revs
> >> COUCHDB-3291: Excessively long document IDs prevent replicator from
> making progress
> >> COUCHDB-3293: Allow limiting length of document ID (for CouchDB
> proper)
> >> COUCHDB-3305: (mango) don’t crash with invalid input to built in
> reducer function
> >> COUCHDB-3362: DELETE attachment on non-existing document creates the
> document, rather than returning 404
> >> COUCHDB-3364: Don’t crash compactor when compacting process fails.
> >> COUCHDB-3367: Require server admin user for db/_compact and
> db_view_cleanup endpoints
> >> COUCHDB-3376: Fix mem3_shards under load
> >> COUCHDB-3378: Fix mango full text detection
> >> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> >> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc
> updated
> >> COUCHDB-3402: race condition in mem3 startup
> >> #511: (mango) Return false for empty list
> >> #595: Return 409 to PUT attachment with non-existent rev
> >> #623: Ensure replicator _active_tasks entry reports recent pending
> changes value
> >> #627: Pass UserCtx to fabric’s all_docs from mango query
> >> #631: fix couchdb_os_proc_pool eunit timeouts
> >> #644: Make couch_event_sup:stop/1 synchronous
> >> #645: Pass db open options to fabric_view_map for _view and _list
> queries on _users DB
> >> #648: Fix couch_replicator_changes_reader:process_change
> >> #649: Avoid a race when restarting an index updater
> >> #667: Prevent a terrible race condition
> >> #677: Make replication filter fetch error for _replicate return a 404
> >> Fix CORS max_age configuration parameter via Access-Control-Max-Age
> >> Chunk missing revisions before attempting to save on target
> (improves replication for very conflicted, very deep revision tree
> documents)
> >> Allow w parameter for attachments
> >> Return “Bad Request” when count in /_uuids exceeds max
> >> Fix crashes when replicator db is deleted
> >> Skip internal replication if changes already replicated
> >> Fix encoding issues on _update/../doc_id and PUT attachments
> >>
> >> Upgrade Notes
> >>
> >> The deprecated (and broken) OAuth 1.0 implementation has been
> removed.
> >> If user code reads or manipulates replicator document states,
> consider using the [replicator] update_docs = true compatibility parameter.
> In that case replicator will continue updating documents with transient
> replication states. However, that will incure a performance cost. Consider
> instead switching using _scheduler/docs HTTP endpoint.
> >> The stale parameter for views and _find has been deprecated in
> favour of two new parameters: stable and update. The old stale=ok behaviour
> is equivalent to stable=true&update=false, and the old stale=update_after
> behaviour is equivalent to stable=true&update=lazy. The deprecated stale
> parameter will be removed in CouchDB 3.0.
> >> The [couchdb] max_document_size configuration parameter was renamed
> to [httpd] max_http_request_size to reflect its actual implementation. It
> has been enhanced by a new [couchdb] single_max_doc_size setting. Both are
> documented in the shipped default.ini file.
> >
>
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by William Edney <be...@technicalpursuit.com>.
Congrats to all, but especially to Joan whose hard work and undying
persistence made this release happen!
As a "basic" user of CouchDB, this release fixed a number of bugs for me
and is going to allow me to use the Couch 2.X series for my clients and
company. If there's ever any doubt to anyone on the committer team that
your efforts make an impact, let me put that to rest - this release is
making a huge difference for me!
Cheers,
- Bill
On Mon, Aug 7, 2017 at 2:14 PM, Benjamin Anderson <b...@banjiewen.net> wrote:
> > but a big and special thanks to Joan Touzet who made it her mission to
> get this done and succeeded.
>
> Indeed. Thanks, Joan!
>
> --
> b
>
> On Mon, Aug 7, 2017 at 11:53 AM, Robert Samuel Newson
> <rn...@apache.org> wrote:
> >
> > Thanks to everyone for the amazing efforts exerted to get here but a big
> and special thanks to Joan Touzet who made it her mission to get this done
> and succeeded.
> >
> > Best,
> > B.
> >
> >> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
> >>
> >> Dear community,
> >>
> >> Apache CouchDB 2.1.0 has been released and is available for
> >> download.
> >>
> >> CouchDB is a database that completely embraces the web. Store your data
> >> with JSON documents. Access your documents with your web browser, via
> >> HTTP. Query, combine, and transform your documents with JavaScript.
> >> CouchDB works well with modern web and mobile apps. You can distribute
> >> your data, efficiently using CouchDB’s incremental replication. CouchDB
> >> supports master-master setups with automatic conflict detection.
> >>
> >> CouchDB comes with a suite of features, such as on-the-fly document
> >> transformation and real-time change notifications, that make web
> >> development a breeze. It even comes with an easy to use web
> >> administration console, served directly out of CouchDB! We care a lot
> >> about distributed scaling. CouchDB is highly available and partition
> >> tolerant, but is also eventually consistent. And we care a lot about
> >> your data. CouchDB has a fault-tolerant storage engine that puts the
> >> safety of your data first.
> >>
> >> Download your copy here:
> >>
> >> https://couchdb.apache.org/#download
> >>
> >> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> >> RHEL/CentOS are available.
> >>
> >> CouchDB 2.1.0 is a feature release, and was originally published on
> >> 2017-08-07.
> >>
> >> The community would like to thank all contributors for their part in
> >> making this release, from the smallest bug report or patch to major
> >> contributions in code, design, or marketing, we couldn’t have done it
> >> without you!
> >>
> >> The release notes follow. A formatted version of them can be viewed
> >> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
> >>
> >> On behalf of the CouchDB PMC,
> >> Joan Touzet
> >>
> >> ---
> >>
> >> Version 2.1.0
> >>
> >> The Mango _find endpoint supports a new combination operator,
> $allMatch, which matches and returns all documents that contain an array
> field with all its elements matching all the specified query criteria.
> >>
> >> New scheduling replicator. The core of the new replicator is a
> scheduler which allows running a large number of replication jobs by
> switching between them, stopping some and starting others periodically.
> Jobs which fail are backed off exponentially. There is also an improved
> inspection and querying API: _scheduler/jobs and _scheduler/docs:
> >> _scheduler/jobs : This endpoint shows active replication jobs.
> These are jobs managed by the scheduler. Some of them might be running,
> some might be waiting to run, or backed off (penalized) because they
> crashed too many times. Semantically this is somewhat equivalent to
> _active_tasks but focuses only on replications. Jobs which have completed
> or which were never created because of malformed replication documents will
> not be shown here as they are not managed by the scheduler. _replicate
> replications, started form _replicate endpoint not from a document in a
> _replicator db, will also show up here.
> >> _scheduler/docs : This endpoint is an improvement on having to
> go back and read replication documents to query their state. It represents
> the state of all the replications started from documents in _replicator db.
> Unlike _scheduler/jobs it will also show jobs which have failed or have
> completed.
> >>
> >> By default, scheduling replicator will not update documents with
> transient states like triggered or error anymore, instead _scheduler/docs
> API should be used to query replication document states.
> >>
> >> Other scheduling replicator improvements
> >>
> >> Network resource usage and performance was improved by
> implementing a shared connection pool. This should help in cases of a large
> number of connections to the same sources or target. Previously connection
> pools were shared only withing a single replication job.
> >> Improved request rate limit handling. Replicator requests will
> auto-discover rate limit capacity on targets and sources based on a proven
> Additive Increase / Multiplicative Decrease feedback control algorithm.
> >> Improved performance by having exponential backoff for all
> replication jobs failures. Previously there were some scenarios were
> failure led to continuous repeated retries, consuming CPU and disk
> resources in the process.
> >> Improved recovery from long but temporary network failure.
> Currently if replications jobs fail to start 10 times in a row, they will
> not be retried anymore. This is sometimes desirable, but in some cases, for
> example, after a sustained DNS failure which eventually recovers,
> replications reach their retry limit, stop retrying and never recover.
> Previously it required user intervention to continue. Scheduling replicator
> will never give up retrying a valid scheduled replication job and so it
> should recover automatically.
> >> Better handling of filtered replications. Failing user filter
> code fetches from the source will not block replicator manager and stall
> other replications. Failing filter fetches will also be backed off
> exponentially. Another improvement is when filter code changes on the
> source, a running replication will detect that and restart itself with a
> new replication ID automatically.
> >>
> >> The 2.1.0 release also includes the following minor improvements:
> >>
> >> COUCHDB-1946: Hibernate couch_stream after each write (up to 70%
> reduction in memory usage during replication of DBs with large attachments)
> >> COUCHDB-2964: Investigate switching replicator manager change feeds
> to using “normal” instead of “longpoll”
> >> COUCHDB-2988: (mango) Allow query selector as changes and
> replication filter
> >> COUCHDB-2992: Add additional support for document size
> >> COUCHDB-3046: Improve reduce function overflow protection
> >> COUCHDB-3061: Use vectored reads to search for buried headers in
> .couch files. “On a modern linux system with SSD, we see improvements up to
> 15x.”
> >> COUCHDB-3063: “stale=ok” option replaced with new “stable” and
> “update” options.
> >> COUCHDB-3180: Add features list in the welcome message
> >> COUCHDB-3203: Make auth handlers configurable (in ini files)
> >> COUCHDB-3234: Track open shard timeouts with a counter instead of
> logging
> >> COUCHDB-3242: Make get view group info timeout in couch_indexer
> configurable
> >> COUCHDB-3249: Add config to disable index all fields (text indexes)
> >> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> >> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and
> binary memory usage
> >> COUCHDB-3298: Optimize writing btree nodes
> >> COUCHDB-3302: (Improve) Attachment replication over low bandwidth
> network connections
> >> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per
> db open
> >> COUCHDB-3318: bypass couch_httpd_vhost if there are none
> >> COUCHDB-3323: Idle dbs cause excessive overhead
> >> COUCHDB-3324: Introduce couch_replicator_scheduler
> >> COUCHDB-3337: End-point _local_docs doesn’t conform to query params
> of _all_docs
> >> COUCHDB-3358: (mango) Use efficient set storage for field names
> >> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit
> configurable
> >> #457: TeX/LaTeX/texinfo removed from default docs build chain
> >> #469: (mango) Choose index based on fields match
> >> #483: couchup database migration tool
> >> #582: Add X-Frame-Options support to help protect against
> clickjacking
> >> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single
> nodes
> >> #624: Enable compaction daemon by default
> >> #626: Allow enable node decom using string “true”
> >> (mango) Configurable default limit, defaults to 25.
> >> (mango) _design documents ignored when querying _all_docs
> >> (mango) add $allMatch selector
> >> Add local.d/default.d directories by default and document
> >> Improved INSTALL.* text files
> >>
> >> Fixed Issues
> >>
> >> The 2.1.0 release includes fixes for the following issues:
> >>
> >> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom
> headers are specified in response of _update handler (missed in 2.0 merge)
> >> COUCHDB-2731: Authentication DB was not considered a system DB
> >> COUCHDB-3010: (Superceded fix for replication exponential backoff)
> >> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers”
> header
> >> COUCHDB-3100: Fix documentation on require_valid_user
> >> COUCHDB-3109: 500 when include_docs=true for linked documents
> >> COUCHDB-3113: fabric:open_revs can return {ok, []}
> >> COUCHDB-3149: Exception written to the log if db deleted while there
> is a change feed running
> >> COUCHDB-3150: Update all shards with stale=update_after
> >> COUCHDB-3158: Fix a crash when connection closes for _update
> >> COUCHDB-3162: Default ssl settings cause a crash
> >> COUCHDB-3164: Request fails when using _changes?feed=eventsource&
> heartbeat=30000
> >> COUCHDB-3168: Replicator doesn’t handle well writing documents to a
> target db which has a small max_document_size
> >> COUCHDB-3173: Views return corrupt data for text fields containing
> non-BMP characters
> >> COUCHDB-3174: max_document_size setting can by bypassed by issuing
> multipart/related requests
> >> COUCHDB-3178: Fabric does not send message when filtering lots of
> documents
> >> COUCHDB-3181: function_clause error when adding attachment to doc in
> _users db
> >> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle
> errors in spawned process
> >> COUCHDB-3193: fabric:open_revs returns multiple results when one of
> the shards has stem_interactive_updates=false
> >> COUCHDB-3199: Replicator VDU function doesn’t acount for an already
> malformed document in replicator db
> >> COUCHDB-3202: (mango) do not allow empty field names
> >> COUCHDB-3220: Handle timeout in _revs_diff
> >> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key
> during document creation
> >> COUCHDB-3231: Allow fixing users’ documents (type and roles)
> >> COUCHDB-3232: user context not passed down in fabric_view_all_docs
> >> COUCHDB-3238: os_process_limit documentation wrong
> >> COUCHDB-3241: race condition in couch_server if delete msg for a db
> is received before open_result msg
> >> COUCHDB-3245: Make couchjs -S option take effect again
> >> COUCHDB-3252: Include main-coffee.js in release artifact (broken
> CoffeeScript view server)
> >> COUCHDB-3255: Conflicts introduced by recreating docs with
> attachments
> >> COUCHDB-3259: Don’t trap exits in couch_file
> >> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> >> COUCHDB-3269: view response can ‘hang’ with filter and limit
> specified
> >> COUCHDB-3271: Replications crash with ‘kaboom’ exit
> >> COUCHDB-3274: eof in couch_file can be incorrect after error
> >> COUCHDB-3277: Replication manager crashes when it finds _replicator
> db shards which are not part of a mem3 db
> >> COUCHDB-3286: Validation function throwing unexpected json crashes
> with function_clause
> >> COUCHDB-3289: handle error clause when calling fabric:open_revs
> >> COUCHDB-3291: Excessively long document IDs prevent replicator from
> making progress
> >> COUCHDB-3293: Allow limiting length of document ID (for CouchDB
> proper)
> >> COUCHDB-3305: (mango) don’t crash with invalid input to built in
> reducer function
> >> COUCHDB-3362: DELETE attachment on non-existing document creates the
> document, rather than returning 404
> >> COUCHDB-3364: Don’t crash compactor when compacting process fails.
> >> COUCHDB-3367: Require server admin user for db/_compact and
> db_view_cleanup endpoints
> >> COUCHDB-3376: Fix mem3_shards under load
> >> COUCHDB-3378: Fix mango full text detection
> >> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> >> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc
> updated
> >> COUCHDB-3402: race condition in mem3 startup
> >> #511: (mango) Return false for empty list
> >> #595: Return 409 to PUT attachment with non-existent rev
> >> #623: Ensure replicator _active_tasks entry reports recent pending
> changes value
> >> #627: Pass UserCtx to fabric’s all_docs from mango query
> >> #631: fix couchdb_os_proc_pool eunit timeouts
> >> #644: Make couch_event_sup:stop/1 synchronous
> >> #645: Pass db open options to fabric_view_map for _view and _list
> queries on _users DB
> >> #648: Fix couch_replicator_changes_reader:process_change
> >> #649: Avoid a race when restarting an index updater
> >> #667: Prevent a terrible race condition
> >> #677: Make replication filter fetch error for _replicate return a 404
> >> Fix CORS max_age configuration parameter via Access-Control-Max-Age
> >> Chunk missing revisions before attempting to save on target
> (improves replication for very conflicted, very deep revision tree
> documents)
> >> Allow w parameter for attachments
> >> Return “Bad Request” when count in /_uuids exceeds max
> >> Fix crashes when replicator db is deleted
> >> Skip internal replication if changes already replicated
> >> Fix encoding issues on _update/../doc_id and PUT attachments
> >>
> >> Upgrade Notes
> >>
> >> The deprecated (and broken) OAuth 1.0 implementation has been
> removed.
> >> If user code reads or manipulates replicator document states,
> consider using the [replicator] update_docs = true compatibility parameter.
> In that case replicator will continue updating documents with transient
> replication states. However, that will incure a performance cost. Consider
> instead switching using _scheduler/docs HTTP endpoint.
> >> The stale parameter for views and _find has been deprecated in
> favour of two new parameters: stable and update. The old stale=ok behaviour
> is equivalent to stable=true&update=false, and the old stale=update_after
> behaviour is equivalent to stable=true&update=lazy. The deprecated stale
> parameter will be removed in CouchDB 3.0.
> >> The [couchdb] max_document_size configuration parameter was renamed
> to [httpd] max_http_request_size to reflect its actual implementation. It
> has been enhanced by a new [couchdb] single_max_doc_size setting. Both are
> documented in the shipped default.ini file.
> >
>
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Benjamin Anderson <b...@banjiewen.net>.
> but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
Indeed. Thanks, Joan!
--
b
On Mon, Aug 7, 2017 at 11:53 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> Thanks to everyone for the amazing efforts exerted to get here but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
>
> Best,
> B.
>
>> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
>>
>> Dear community,
>>
>> Apache CouchDB 2.1.0 has been released and is available for
>> download.
>>
>> CouchDB is a database that completely embraces the web. Store your data
>> with JSON documents. Access your documents with your web browser, via
>> HTTP. Query, combine, and transform your documents with JavaScript.
>> CouchDB works well with modern web and mobile apps. You can distribute
>> your data, efficiently using CouchDB’s incremental replication. CouchDB
>> supports master-master setups with automatic conflict detection.
>>
>> CouchDB comes with a suite of features, such as on-the-fly document
>> transformation and real-time change notifications, that make web
>> development a breeze. It even comes with an easy to use web
>> administration console, served directly out of CouchDB! We care a lot
>> about distributed scaling. CouchDB is highly available and partition
>> tolerant, but is also eventually consistent. And we care a lot about
>> your data. CouchDB has a fault-tolerant storage engine that puts the
>> safety of your data first.
>>
>> Download your copy here:
>>
>> https://couchdb.apache.org/#download
>>
>> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
>> RHEL/CentOS are available.
>>
>> CouchDB 2.1.0 is a feature release, and was originally published on
>> 2017-08-07.
>>
>> The community would like to thank all contributors for their part in
>> making this release, from the smallest bug report or patch to major
>> contributions in code, design, or marketing, we couldn’t have done it
>> without you!
>>
>> The release notes follow. A formatted version of them can be viewed
>> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
>>
>> On behalf of the CouchDB PMC,
>> Joan Touzet
>>
>> ---
>>
>> Version 2.1.0
>>
>> The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
>>
>> New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
>> _scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
>> _scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
>>
>> By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
>>
>> Other scheduling replicator improvements
>>
>> Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
>> Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
>> Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
>> Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
>> Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
>>
>> The 2.1.0 release also includes the following minor improvements:
>>
>> COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
>> COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
>> COUCHDB-2988: (mango) Allow query selector as changes and replication filter
>> COUCHDB-2992: Add additional support for document size
>> COUCHDB-3046: Improve reduce function overflow protection
>> COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
>> COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
>> COUCHDB-3180: Add features list in the welcome message
>> COUCHDB-3203: Make auth handlers configurable (in ini files)
>> COUCHDB-3234: Track open shard timeouts with a counter instead of logging
>> COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
>> COUCHDB-3249: Add config to disable index all fields (text indexes)
>> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
>> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
>> COUCHDB-3298: Optimize writing btree nodes
>> COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
>> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
>> COUCHDB-3318: bypass couch_httpd_vhost if there are none
>> COUCHDB-3323: Idle dbs cause excessive overhead
>> COUCHDB-3324: Introduce couch_replicator_scheduler
>> COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
>> COUCHDB-3358: (mango) Use efficient set storage for field names
>> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
>> #457: TeX/LaTeX/texinfo removed from default docs build chain
>> #469: (mango) Choose index based on fields match
>> #483: couchup database migration tool
>> #582: Add X-Frame-Options support to help protect against clickjacking
>> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
>> #624: Enable compaction daemon by default
>> #626: Allow enable node decom using string “true”
>> (mango) Configurable default limit, defaults to 25.
>> (mango) _design documents ignored when querying _all_docs
>> (mango) add $allMatch selector
>> Add local.d/default.d directories by default and document
>> Improved INSTALL.* text files
>>
>> Fixed Issues
>>
>> The 2.1.0 release includes fixes for the following issues:
>>
>> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
>> COUCHDB-2731: Authentication DB was not considered a system DB
>> COUCHDB-3010: (Superceded fix for replication exponential backoff)
>> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
>> COUCHDB-3100: Fix documentation on require_valid_user
>> COUCHDB-3109: 500 when include_docs=true for linked documents
>> COUCHDB-3113: fabric:open_revs can return {ok, []}
>> COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
>> COUCHDB-3150: Update all shards with stale=update_after
>> COUCHDB-3158: Fix a crash when connection closes for _update
>> COUCHDB-3162: Default ssl settings cause a crash
>> COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
>> COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
>> COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
>> COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
>> COUCHDB-3178: Fabric does not send message when filtering lots of documents
>> COUCHDB-3181: function_clause error when adding attachment to doc in _users db
>> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
>> COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
>> COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
>> COUCHDB-3202: (mango) do not allow empty field names
>> COUCHDB-3220: Handle timeout in _revs_diff
>> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
>> COUCHDB-3231: Allow fixing users’ documents (type and roles)
>> COUCHDB-3232: user context not passed down in fabric_view_all_docs
>> COUCHDB-3238: os_process_limit documentation wrong
>> COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
>> COUCHDB-3245: Make couchjs -S option take effect again
>> COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
>> COUCHDB-3255: Conflicts introduced by recreating docs with attachments
>> COUCHDB-3259: Don’t trap exits in couch_file
>> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
>> COUCHDB-3269: view response can ‘hang’ with filter and limit specified
>> COUCHDB-3271: Replications crash with ‘kaboom’ exit
>> COUCHDB-3274: eof in couch_file can be incorrect after error
>> COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
>> COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
>> COUCHDB-3289: handle error clause when calling fabric:open_revs
>> COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
>> COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
>> COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
>> COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
>> COUCHDB-3364: Don’t crash compactor when compacting process fails.
>> COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
>> COUCHDB-3376: Fix mem3_shards under load
>> COUCHDB-3378: Fix mango full text detection
>> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
>> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
>> COUCHDB-3402: race condition in mem3 startup
>> #511: (mango) Return false for empty list
>> #595: Return 409 to PUT attachment with non-existent rev
>> #623: Ensure replicator _active_tasks entry reports recent pending changes value
>> #627: Pass UserCtx to fabric’s all_docs from mango query
>> #631: fix couchdb_os_proc_pool eunit timeouts
>> #644: Make couch_event_sup:stop/1 synchronous
>> #645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
>> #648: Fix couch_replicator_changes_reader:process_change
>> #649: Avoid a race when restarting an index updater
>> #667: Prevent a terrible race condition
>> #677: Make replication filter fetch error for _replicate return a 404
>> Fix CORS max_age configuration parameter via Access-Control-Max-Age
>> Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
>> Allow w parameter for attachments
>> Return “Bad Request” when count in /_uuids exceeds max
>> Fix crashes when replicator db is deleted
>> Skip internal replication if changes already replicated
>> Fix encoding issues on _update/../doc_id and PUT attachments
>>
>> Upgrade Notes
>>
>> The deprecated (and broken) OAuth 1.0 implementation has been removed.
>> If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
>> The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
>> The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.
>
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Benjamin Anderson <b...@banjiewen.net>.
> but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
Indeed. Thanks, Joan!
--
b
On Mon, Aug 7, 2017 at 11:53 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> Thanks to everyone for the amazing efforts exerted to get here but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
>
> Best,
> B.
>
>> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
>>
>> Dear community,
>>
>> Apache CouchDB 2.1.0 has been released and is available for
>> download.
>>
>> CouchDB is a database that completely embraces the web. Store your data
>> with JSON documents. Access your documents with your web browser, via
>> HTTP. Query, combine, and transform your documents with JavaScript.
>> CouchDB works well with modern web and mobile apps. You can distribute
>> your data, efficiently using CouchDB’s incremental replication. CouchDB
>> supports master-master setups with automatic conflict detection.
>>
>> CouchDB comes with a suite of features, such as on-the-fly document
>> transformation and real-time change notifications, that make web
>> development a breeze. It even comes with an easy to use web
>> administration console, served directly out of CouchDB! We care a lot
>> about distributed scaling. CouchDB is highly available and partition
>> tolerant, but is also eventually consistent. And we care a lot about
>> your data. CouchDB has a fault-tolerant storage engine that puts the
>> safety of your data first.
>>
>> Download your copy here:
>>
>> https://couchdb.apache.org/#download
>>
>> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
>> RHEL/CentOS are available.
>>
>> CouchDB 2.1.0 is a feature release, and was originally published on
>> 2017-08-07.
>>
>> The community would like to thank all contributors for their part in
>> making this release, from the smallest bug report or patch to major
>> contributions in code, design, or marketing, we couldn’t have done it
>> without you!
>>
>> The release notes follow. A formatted version of them can be viewed
>> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
>>
>> On behalf of the CouchDB PMC,
>> Joan Touzet
>>
>> ---
>>
>> Version 2.1.0
>>
>> The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
>>
>> New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
>> _scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
>> _scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
>>
>> By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
>>
>> Other scheduling replicator improvements
>>
>> Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
>> Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
>> Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
>> Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
>> Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
>>
>> The 2.1.0 release also includes the following minor improvements:
>>
>> COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
>> COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
>> COUCHDB-2988: (mango) Allow query selector as changes and replication filter
>> COUCHDB-2992: Add additional support for document size
>> COUCHDB-3046: Improve reduce function overflow protection
>> COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
>> COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
>> COUCHDB-3180: Add features list in the welcome message
>> COUCHDB-3203: Make auth handlers configurable (in ini files)
>> COUCHDB-3234: Track open shard timeouts with a counter instead of logging
>> COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
>> COUCHDB-3249: Add config to disable index all fields (text indexes)
>> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
>> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
>> COUCHDB-3298: Optimize writing btree nodes
>> COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
>> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
>> COUCHDB-3318: bypass couch_httpd_vhost if there are none
>> COUCHDB-3323: Idle dbs cause excessive overhead
>> COUCHDB-3324: Introduce couch_replicator_scheduler
>> COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
>> COUCHDB-3358: (mango) Use efficient set storage for field names
>> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
>> #457: TeX/LaTeX/texinfo removed from default docs build chain
>> #469: (mango) Choose index based on fields match
>> #483: couchup database migration tool
>> #582: Add X-Frame-Options support to help protect against clickjacking
>> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
>> #624: Enable compaction daemon by default
>> #626: Allow enable node decom using string “true”
>> (mango) Configurable default limit, defaults to 25.
>> (mango) _design documents ignored when querying _all_docs
>> (mango) add $allMatch selector
>> Add local.d/default.d directories by default and document
>> Improved INSTALL.* text files
>>
>> Fixed Issues
>>
>> The 2.1.0 release includes fixes for the following issues:
>>
>> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
>> COUCHDB-2731: Authentication DB was not considered a system DB
>> COUCHDB-3010: (Superceded fix for replication exponential backoff)
>> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
>> COUCHDB-3100: Fix documentation on require_valid_user
>> COUCHDB-3109: 500 when include_docs=true for linked documents
>> COUCHDB-3113: fabric:open_revs can return {ok, []}
>> COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
>> COUCHDB-3150: Update all shards with stale=update_after
>> COUCHDB-3158: Fix a crash when connection closes for _update
>> COUCHDB-3162: Default ssl settings cause a crash
>> COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
>> COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
>> COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
>> COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
>> COUCHDB-3178: Fabric does not send message when filtering lots of documents
>> COUCHDB-3181: function_clause error when adding attachment to doc in _users db
>> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
>> COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
>> COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
>> COUCHDB-3202: (mango) do not allow empty field names
>> COUCHDB-3220: Handle timeout in _revs_diff
>> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
>> COUCHDB-3231: Allow fixing users’ documents (type and roles)
>> COUCHDB-3232: user context not passed down in fabric_view_all_docs
>> COUCHDB-3238: os_process_limit documentation wrong
>> COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
>> COUCHDB-3245: Make couchjs -S option take effect again
>> COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
>> COUCHDB-3255: Conflicts introduced by recreating docs with attachments
>> COUCHDB-3259: Don’t trap exits in couch_file
>> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
>> COUCHDB-3269: view response can ‘hang’ with filter and limit specified
>> COUCHDB-3271: Replications crash with ‘kaboom’ exit
>> COUCHDB-3274: eof in couch_file can be incorrect after error
>> COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
>> COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
>> COUCHDB-3289: handle error clause when calling fabric:open_revs
>> COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
>> COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
>> COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
>> COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
>> COUCHDB-3364: Don’t crash compactor when compacting process fails.
>> COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
>> COUCHDB-3376: Fix mem3_shards under load
>> COUCHDB-3378: Fix mango full text detection
>> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
>> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
>> COUCHDB-3402: race condition in mem3 startup
>> #511: (mango) Return false for empty list
>> #595: Return 409 to PUT attachment with non-existent rev
>> #623: Ensure replicator _active_tasks entry reports recent pending changes value
>> #627: Pass UserCtx to fabric’s all_docs from mango query
>> #631: fix couchdb_os_proc_pool eunit timeouts
>> #644: Make couch_event_sup:stop/1 synchronous
>> #645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
>> #648: Fix couch_replicator_changes_reader:process_change
>> #649: Avoid a race when restarting an index updater
>> #667: Prevent a terrible race condition
>> #677: Make replication filter fetch error for _replicate return a 404
>> Fix CORS max_age configuration parameter via Access-Control-Max-Age
>> Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
>> Allow w parameter for attachments
>> Return “Bad Request” when count in /_uuids exceeds max
>> Fix crashes when replicator db is deleted
>> Skip internal replication if changes already replicated
>> Fix encoding issues on _update/../doc_id and PUT attachments
>>
>> Upgrade Notes
>>
>> The deprecated (and broken) OAuth 1.0 implementation has been removed.
>> If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
>> The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
>> The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.
>
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Geoffrey Cox <re...@gmail.com>.
This is awesome!
On Mon, Aug 7, 2017, 11:53 Robert Samuel Newson <rn...@apache.org> wrote:
>
> Thanks to everyone for the amazing efforts exerted to get here but a big
> and special thanks to Joan Touzet who made it her mission to get this done
> and succeeded.
>
> Best,
> B.
>
> > On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
> >
> > Dear community,
> >
> > Apache CouchDB 2.1.0 has been released and is available for
> > download.
> >
> > CouchDB is a database that completely embraces the web. Store your data
> > with JSON documents. Access your documents with your web browser, via
> > HTTP. Query, combine, and transform your documents with JavaScript.
> > CouchDB works well with modern web and mobile apps. You can distribute
> > your data, efficiently using CouchDB’s incremental replication. CouchDB
> > supports master-master setups with automatic conflict detection.
> >
> > CouchDB comes with a suite of features, such as on-the-fly document
> > transformation and real-time change notifications, that make web
> > development a breeze. It even comes with an easy to use web
> > administration console, served directly out of CouchDB! We care a lot
> > about distributed scaling. CouchDB is highly available and partition
> > tolerant, but is also eventually consistent. And we care a lot about
> > your data. CouchDB has a fault-tolerant storage engine that puts the
> > safety of your data first.
> >
> > Download your copy here:
> >
> > https://couchdb.apache.org/#download
> >
> > Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> > RHEL/CentOS are available.
> >
> > CouchDB 2.1.0 is a feature release, and was originally published on
> > 2017-08-07.
> >
> > The community would like to thank all contributors for their part in
> > making this release, from the smallest bug report or patch to major
> > contributions in code, design, or marketing, we couldn’t have done it
> > without you!
> >
> > The release notes follow. A formatted version of them can be viewed
> > at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
> >
> > On behalf of the CouchDB PMC,
> > Joan Touzet
> >
> > ---
> >
> > Version 2.1.0
> >
> > The Mango _find endpoint supports a new combination operator,
> $allMatch, which matches and returns all documents that contain an array
> field with all its elements matching all the specified query criteria.
> >
> > New scheduling replicator. The core of the new replicator is a
> scheduler which allows running a large number of replication jobs by
> switching between them, stopping some and starting others periodically.
> Jobs which fail are backed off exponentially. There is also an improved
> inspection and querying API: _scheduler/jobs and _scheduler/docs:
> > _scheduler/jobs : This endpoint shows active replication jobs.
> These are jobs managed by the scheduler. Some of them might be running,
> some might be waiting to run, or backed off (penalized) because they
> crashed too many times. Semantically this is somewhat equivalent to
> _active_tasks but focuses only on replications. Jobs which have completed
> or which were never created because of malformed replication documents will
> not be shown here as they are not managed by the scheduler. _replicate
> replications, started form _replicate endpoint not from a document in a
> _replicator db, will also show up here.
> > _scheduler/docs : This endpoint is an improvement on having to go
> back and read replication documents to query their state. It represents the
> state of all the replications started from documents in _replicator db.
> Unlike _scheduler/jobs it will also show jobs which have failed or have
> completed.
> >
> > By default, scheduling replicator will not update documents with
> transient states like triggered or error anymore, instead _scheduler/docs
> API should be used to query replication document states.
> >
> > Other scheduling replicator improvements
> >
> > Network resource usage and performance was improved by
> implementing a shared connection pool. This should help in cases of a large
> number of connections to the same sources or target. Previously connection
> pools were shared only withing a single replication job.
> > Improved request rate limit handling. Replicator requests will
> auto-discover rate limit capacity on targets and sources based on a proven
> Additive Increase / Multiplicative Decrease feedback control algorithm.
> > Improved performance by having exponential backoff for all
> replication jobs failures. Previously there were some scenarios were
> failure led to continuous repeated retries, consuming CPU and disk
> resources in the process.
> > Improved recovery from long but temporary network failure.
> Currently if replications jobs fail to start 10 times in a row, they will
> not be retried anymore. This is sometimes desirable, but in some cases, for
> example, after a sustained DNS failure which eventually recovers,
> replications reach their retry limit, stop retrying and never recover.
> Previously it required user intervention to continue. Scheduling replicator
> will never give up retrying a valid scheduled replication job and so it
> should recover automatically.
> > Better handling of filtered replications. Failing user filter
> code fetches from the source will not block replicator manager and stall
> other replications. Failing filter fetches will also be backed off
> exponentially. Another improvement is when filter code changes on the
> source, a running replication will detect that and restart itself with a
> new replication ID automatically.
> >
> > The 2.1.0 release also includes the following minor improvements:
> >
> > COUCHDB-1946: Hibernate couch_stream after each write (up to 70%
> reduction in memory usage during replication of DBs with large attachments)
> > COUCHDB-2964: Investigate switching replicator manager change feeds
> to using “normal” instead of “longpoll”
> > COUCHDB-2988: (mango) Allow query selector as changes and replication
> filter
> > COUCHDB-2992: Add additional support for document size
> > COUCHDB-3046: Improve reduce function overflow protection
> > COUCHDB-3061: Use vectored reads to search for buried headers in
> .couch files. “On a modern linux system with SSD, we see improvements up to
> 15x.”
> > COUCHDB-3063: “stale=ok” option replaced with new “stable” and
> “update” options.
> > COUCHDB-3180: Add features list in the welcome message
> > COUCHDB-3203: Make auth handlers configurable (in ini files)
> > COUCHDB-3234: Track open shard timeouts with a counter instead of
> logging
> > COUCHDB-3242: Make get view group info timeout in couch_indexer
> configurable
> > COUCHDB-3249: Add config to disable index all fields (text indexes)
> > COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> > COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary
> memory usage
> > COUCHDB-3298: Optimize writing btree nodes
> > COUCHDB-3302: (Improve) Attachment replication over low bandwidth
> network connections
> > COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per
> db open
> > COUCHDB-3318: bypass couch_httpd_vhost if there are none
> > COUCHDB-3323: Idle dbs cause excessive overhead
> > COUCHDB-3324: Introduce couch_replicator_scheduler
> > COUCHDB-3337: End-point _local_docs doesn’t conform to query params
> of _all_docs
> > COUCHDB-3358: (mango) Use efficient set storage for field names
> > COUCHDB-3425: Make _doc_ids _changes filter fast-path limit
> configurable
> > #457: TeX/LaTeX/texinfo removed from default docs build chain
> > #469: (mango) Choose index based on fields match
> > #483: couchup database migration tool
> > #582: Add X-Frame-Options support to help protect against clickjacking
> > #593: Allow bind address of 127.0.0.1 in _cluster_setup for single
> nodes
> > #624: Enable compaction daemon by default
> > #626: Allow enable node decom using string “true”
> > (mango) Configurable default limit, defaults to 25.
> > (mango) _design documents ignored when querying _all_docs
> > (mango) add $allMatch selector
> > Add local.d/default.d directories by default and document
> > Improved INSTALL.* text files
> >
> > Fixed Issues
> >
> > The 2.1.0 release includes fixes for the following issues:
> >
> > COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom
> headers are specified in response of _update handler (missed in 2.0 merge)
> > COUCHDB-2731: Authentication DB was not considered a system DB
> > COUCHDB-3010: (Superceded fix for replication exponential backoff)
> > COUCHDB-3090: Error when handling empty
> “Access-Control-Request-Headers” header
> > COUCHDB-3100: Fix documentation on require_valid_user
> > COUCHDB-3109: 500 when include_docs=true for linked documents
> > COUCHDB-3113: fabric:open_revs can return {ok, []}
> > COUCHDB-3149: Exception written to the log if db deleted while there
> is a change feed running
> > COUCHDB-3150: Update all shards with stale=update_after
> > COUCHDB-3158: Fix a crash when connection closes for _update
> > COUCHDB-3162: Default ssl settings cause a crash
> > COUCHDB-3164: Request fails when using
> _changes?feed=eventsource&heartbeat=30000
> > COUCHDB-3168: Replicator doesn’t handle well writing documents to a
> target db which has a small max_document_size
> > COUCHDB-3173: Views return corrupt data for text fields containing
> non-BMP characters
> > COUCHDB-3174: max_document_size setting can by bypassed by issuing
> multipart/related requests
> > COUCHDB-3178: Fabric does not send message when filtering lots of
> documents
> > COUCHDB-3181: function_clause error when adding attachment to doc in
> _users db
> > COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle
> errors in spawned process
> > COUCHDB-3193: fabric:open_revs returns multiple results when one of
> the shards has stem_interactive_updates=false
> > COUCHDB-3199: Replicator VDU function doesn’t acount for an already
> malformed document in replicator db
> > COUCHDB-3202: (mango) do not allow empty field names
> > COUCHDB-3220: Handle timeout in _revs_diff
> > COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key
> during document creation
> > COUCHDB-3231: Allow fixing users’ documents (type and roles)
> > COUCHDB-3232: user context not passed down in fabric_view_all_docs
> > COUCHDB-3238: os_process_limit documentation wrong
> > COUCHDB-3241: race condition in couch_server if delete msg for a db
> is received before open_result msg
> > COUCHDB-3245: Make couchjs -S option take effect again
> > COUCHDB-3252: Include main-coffee.js in release artifact (broken
> CoffeeScript view server)
> > COUCHDB-3255: Conflicts introduced by recreating docs with attachments
> > COUCHDB-3259: Don’t trap exits in couch_file
> > COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> > COUCHDB-3269: view response can ‘hang’ with filter and limit specified
> > COUCHDB-3271: Replications crash with ‘kaboom’ exit
> > COUCHDB-3274: eof in couch_file can be incorrect after error
> > COUCHDB-3277: Replication manager crashes when it finds _replicator
> db shards which are not part of a mem3 db
> > COUCHDB-3286: Validation function throwing unexpected json crashes
> with function_clause
> > COUCHDB-3289: handle error clause when calling fabric:open_revs
> > COUCHDB-3291: Excessively long document IDs prevent replicator from
> making progress
> > COUCHDB-3293: Allow limiting length of document ID (for CouchDB
> proper)
> > COUCHDB-3305: (mango) don’t crash with invalid input to built in
> reducer function
> > COUCHDB-3362: DELETE attachment on non-existing document creates the
> document, rather than returning 404
> > COUCHDB-3364: Don’t crash compactor when compacting process fails.
> > COUCHDB-3367: Require server admin user for db/_compact and
> db_view_cleanup endpoints
> > COUCHDB-3376: Fix mem3_shards under load
> > COUCHDB-3378: Fix mango full text detection
> > COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> > COUCHDB-3400: Notify couch_index_processes on all shards when ddoc
> updated
> > COUCHDB-3402: race condition in mem3 startup
> > #511: (mango) Return false for empty list
> > #595: Return 409 to PUT attachment with non-existent rev
> > #623: Ensure replicator _active_tasks entry reports recent pending
> changes value
> > #627: Pass UserCtx to fabric’s all_docs from mango query
> > #631: fix couchdb_os_proc_pool eunit timeouts
> > #644: Make couch_event_sup:stop/1 synchronous
> > #645: Pass db open options to fabric_view_map for _view and _list
> queries on _users DB
> > #648: Fix couch_replicator_changes_reader:process_change
> > #649: Avoid a race when restarting an index updater
> > #667: Prevent a terrible race condition
> > #677: Make replication filter fetch error for _replicate return a 404
> > Fix CORS max_age configuration parameter via Access-Control-Max-Age
> > Chunk missing revisions before attempting to save on target (improves
> replication for very conflicted, very deep revision tree documents)
> > Allow w parameter for attachments
> > Return “Bad Request” when count in /_uuids exceeds max
> > Fix crashes when replicator db is deleted
> > Skip internal replication if changes already replicated
> > Fix encoding issues on _update/../doc_id and PUT attachments
> >
> > Upgrade Notes
> >
> > The deprecated (and broken) OAuth 1.0 implementation has been removed.
> > If user code reads or manipulates replicator document states,
> consider using the [replicator] update_docs = true compatibility parameter.
> In that case replicator will continue updating documents with transient
> replication states. However, that will incure a performance cost. Consider
> instead switching using _scheduler/docs HTTP endpoint.
> > The stale parameter for views and _find has been deprecated in favour
> of two new parameters: stable and update. The old stale=ok behaviour is
> equivalent to stable=true&update=false, and the old stale=update_after
> behaviour is equivalent to stable=true&update=lazy. The deprecated stale
> parameter will be removed in CouchDB 3.0.
> > The [couchdb] max_document_size configuration parameter was renamed
> to [httpd] max_http_request_size to reflect its actual implementation. It
> has been enhanced by a new [couchdb] single_max_doc_size setting. Both are
> documented in the shipped default.ini file.
>
>
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Benjamin Anderson <b...@banjiewen.net>.
> but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
Indeed. Thanks, Joan!
--
b
On Mon, Aug 7, 2017 at 11:53 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> Thanks to everyone for the amazing efforts exerted to get here but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
>
> Best,
> B.
>
>> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
>>
>> Dear community,
>>
>> Apache CouchDB 2.1.0 has been released and is available for
>> download.
>>
>> CouchDB is a database that completely embraces the web. Store your data
>> with JSON documents. Access your documents with your web browser, via
>> HTTP. Query, combine, and transform your documents with JavaScript.
>> CouchDB works well with modern web and mobile apps. You can distribute
>> your data, efficiently using CouchDB’s incremental replication. CouchDB
>> supports master-master setups with automatic conflict detection.
>>
>> CouchDB comes with a suite of features, such as on-the-fly document
>> transformation and real-time change notifications, that make web
>> development a breeze. It even comes with an easy to use web
>> administration console, served directly out of CouchDB! We care a lot
>> about distributed scaling. CouchDB is highly available and partition
>> tolerant, but is also eventually consistent. And we care a lot about
>> your data. CouchDB has a fault-tolerant storage engine that puts the
>> safety of your data first.
>>
>> Download your copy here:
>>
>> https://couchdb.apache.org/#download
>>
>> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
>> RHEL/CentOS are available.
>>
>> CouchDB 2.1.0 is a feature release, and was originally published on
>> 2017-08-07.
>>
>> The community would like to thank all contributors for their part in
>> making this release, from the smallest bug report or patch to major
>> contributions in code, design, or marketing, we couldn’t have done it
>> without you!
>>
>> The release notes follow. A formatted version of them can be viewed
>> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
>>
>> On behalf of the CouchDB PMC,
>> Joan Touzet
>>
>> ---
>>
>> Version 2.1.0
>>
>> The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
>>
>> New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
>> _scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
>> _scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
>>
>> By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
>>
>> Other scheduling replicator improvements
>>
>> Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
>> Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
>> Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
>> Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
>> Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
>>
>> The 2.1.0 release also includes the following minor improvements:
>>
>> COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
>> COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
>> COUCHDB-2988: (mango) Allow query selector as changes and replication filter
>> COUCHDB-2992: Add additional support for document size
>> COUCHDB-3046: Improve reduce function overflow protection
>> COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
>> COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
>> COUCHDB-3180: Add features list in the welcome message
>> COUCHDB-3203: Make auth handlers configurable (in ini files)
>> COUCHDB-3234: Track open shard timeouts with a counter instead of logging
>> COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
>> COUCHDB-3249: Add config to disable index all fields (text indexes)
>> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
>> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
>> COUCHDB-3298: Optimize writing btree nodes
>> COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
>> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
>> COUCHDB-3318: bypass couch_httpd_vhost if there are none
>> COUCHDB-3323: Idle dbs cause excessive overhead
>> COUCHDB-3324: Introduce couch_replicator_scheduler
>> COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
>> COUCHDB-3358: (mango) Use efficient set storage for field names
>> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
>> #457: TeX/LaTeX/texinfo removed from default docs build chain
>> #469: (mango) Choose index based on fields match
>> #483: couchup database migration tool
>> #582: Add X-Frame-Options support to help protect against clickjacking
>> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
>> #624: Enable compaction daemon by default
>> #626: Allow enable node decom using string “true”
>> (mango) Configurable default limit, defaults to 25.
>> (mango) _design documents ignored when querying _all_docs
>> (mango) add $allMatch selector
>> Add local.d/default.d directories by default and document
>> Improved INSTALL.* text files
>>
>> Fixed Issues
>>
>> The 2.1.0 release includes fixes for the following issues:
>>
>> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
>> COUCHDB-2731: Authentication DB was not considered a system DB
>> COUCHDB-3010: (Superceded fix for replication exponential backoff)
>> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
>> COUCHDB-3100: Fix documentation on require_valid_user
>> COUCHDB-3109: 500 when include_docs=true for linked documents
>> COUCHDB-3113: fabric:open_revs can return {ok, []}
>> COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
>> COUCHDB-3150: Update all shards with stale=update_after
>> COUCHDB-3158: Fix a crash when connection closes for _update
>> COUCHDB-3162: Default ssl settings cause a crash
>> COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
>> COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
>> COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
>> COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
>> COUCHDB-3178: Fabric does not send message when filtering lots of documents
>> COUCHDB-3181: function_clause error when adding attachment to doc in _users db
>> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
>> COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
>> COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
>> COUCHDB-3202: (mango) do not allow empty field names
>> COUCHDB-3220: Handle timeout in _revs_diff
>> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
>> COUCHDB-3231: Allow fixing users’ documents (type and roles)
>> COUCHDB-3232: user context not passed down in fabric_view_all_docs
>> COUCHDB-3238: os_process_limit documentation wrong
>> COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
>> COUCHDB-3245: Make couchjs -S option take effect again
>> COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
>> COUCHDB-3255: Conflicts introduced by recreating docs with attachments
>> COUCHDB-3259: Don’t trap exits in couch_file
>> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
>> COUCHDB-3269: view response can ‘hang’ with filter and limit specified
>> COUCHDB-3271: Replications crash with ‘kaboom’ exit
>> COUCHDB-3274: eof in couch_file can be incorrect after error
>> COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
>> COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
>> COUCHDB-3289: handle error clause when calling fabric:open_revs
>> COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
>> COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
>> COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
>> COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
>> COUCHDB-3364: Don’t crash compactor when compacting process fails.
>> COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
>> COUCHDB-3376: Fix mem3_shards under load
>> COUCHDB-3378: Fix mango full text detection
>> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
>> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
>> COUCHDB-3402: race condition in mem3 startup
>> #511: (mango) Return false for empty list
>> #595: Return 409 to PUT attachment with non-existent rev
>> #623: Ensure replicator _active_tasks entry reports recent pending changes value
>> #627: Pass UserCtx to fabric’s all_docs from mango query
>> #631: fix couchdb_os_proc_pool eunit timeouts
>> #644: Make couch_event_sup:stop/1 synchronous
>> #645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
>> #648: Fix couch_replicator_changes_reader:process_change
>> #649: Avoid a race when restarting an index updater
>> #667: Prevent a terrible race condition
>> #677: Make replication filter fetch error for _replicate return a 404
>> Fix CORS max_age configuration parameter via Access-Control-Max-Age
>> Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
>> Allow w parameter for attachments
>> Return “Bad Request” when count in /_uuids exceeds max
>> Fix crashes when replicator db is deleted
>> Skip internal replication if changes already replicated
>> Fix encoding issues on _update/../doc_id and PUT attachments
>>
>> Upgrade Notes
>>
>> The deprecated (and broken) OAuth 1.0 implementation has been removed.
>> If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
>> The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
>> The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.
>
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Robert Samuel Newson <rn...@apache.org>.
Thanks to everyone for the amazing efforts exerted to get here but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
Best,
B.
> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
>
> Dear community,
>
> Apache CouchDB 2.1.0 has been released and is available for
> download.
>
> CouchDB is a database that completely embraces the web. Store your data
> with JSON documents. Access your documents with your web browser, via
> HTTP. Query, combine, and transform your documents with JavaScript.
> CouchDB works well with modern web and mobile apps. You can distribute
> your data, efficiently using CouchDB’s incremental replication. CouchDB
> supports master-master setups with automatic conflict detection.
>
> CouchDB comes with a suite of features, such as on-the-fly document
> transformation and real-time change notifications, that make web
> development a breeze. It even comes with an easy to use web
> administration console, served directly out of CouchDB! We care a lot
> about distributed scaling. CouchDB is highly available and partition
> tolerant, but is also eventually consistent. And we care a lot about
> your data. CouchDB has a fault-tolerant storage engine that puts the
> safety of your data first.
>
> Download your copy here:
>
> https://couchdb.apache.org/#download
>
> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> RHEL/CentOS are available.
>
> CouchDB 2.1.0 is a feature release, and was originally published on
> 2017-08-07.
>
> The community would like to thank all contributors for their part in
> making this release, from the smallest bug report or patch to major
> contributions in code, design, or marketing, we couldn’t have done it
> without you!
>
> The release notes follow. A formatted version of them can be viewed
> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
>
> On behalf of the CouchDB PMC,
> Joan Touzet
>
> ---
>
> Version 2.1.0
>
> The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
>
> New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
> _scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
> _scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
>
> By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
>
> Other scheduling replicator improvements
>
> Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
> Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
> Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
> Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
> Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
>
> The 2.1.0 release also includes the following minor improvements:
>
> COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
> COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
> COUCHDB-2988: (mango) Allow query selector as changes and replication filter
> COUCHDB-2992: Add additional support for document size
> COUCHDB-3046: Improve reduce function overflow protection
> COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
> COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
> COUCHDB-3180: Add features list in the welcome message
> COUCHDB-3203: Make auth handlers configurable (in ini files)
> COUCHDB-3234: Track open shard timeouts with a counter instead of logging
> COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
> COUCHDB-3249: Add config to disable index all fields (text indexes)
> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
> COUCHDB-3298: Optimize writing btree nodes
> COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
> COUCHDB-3318: bypass couch_httpd_vhost if there are none
> COUCHDB-3323: Idle dbs cause excessive overhead
> COUCHDB-3324: Introduce couch_replicator_scheduler
> COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
> COUCHDB-3358: (mango) Use efficient set storage for field names
> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
> #457: TeX/LaTeX/texinfo removed from default docs build chain
> #469: (mango) Choose index based on fields match
> #483: couchup database migration tool
> #582: Add X-Frame-Options support to help protect against clickjacking
> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
> #624: Enable compaction daemon by default
> #626: Allow enable node decom using string “true”
> (mango) Configurable default limit, defaults to 25.
> (mango) _design documents ignored when querying _all_docs
> (mango) add $allMatch selector
> Add local.d/default.d directories by default and document
> Improved INSTALL.* text files
>
> Fixed Issues
>
> The 2.1.0 release includes fixes for the following issues:
>
> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
> COUCHDB-2731: Authentication DB was not considered a system DB
> COUCHDB-3010: (Superceded fix for replication exponential backoff)
> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
> COUCHDB-3100: Fix documentation on require_valid_user
> COUCHDB-3109: 500 when include_docs=true for linked documents
> COUCHDB-3113: fabric:open_revs can return {ok, []}
> COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
> COUCHDB-3150: Update all shards with stale=update_after
> COUCHDB-3158: Fix a crash when connection closes for _update
> COUCHDB-3162: Default ssl settings cause a crash
> COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
> COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
> COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
> COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
> COUCHDB-3178: Fabric does not send message when filtering lots of documents
> COUCHDB-3181: function_clause error when adding attachment to doc in _users db
> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
> COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
> COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
> COUCHDB-3202: (mango) do not allow empty field names
> COUCHDB-3220: Handle timeout in _revs_diff
> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
> COUCHDB-3231: Allow fixing users’ documents (type and roles)
> COUCHDB-3232: user context not passed down in fabric_view_all_docs
> COUCHDB-3238: os_process_limit documentation wrong
> COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
> COUCHDB-3245: Make couchjs -S option take effect again
> COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
> COUCHDB-3255: Conflicts introduced by recreating docs with attachments
> COUCHDB-3259: Don’t trap exits in couch_file
> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> COUCHDB-3269: view response can ‘hang’ with filter and limit specified
> COUCHDB-3271: Replications crash with ‘kaboom’ exit
> COUCHDB-3274: eof in couch_file can be incorrect after error
> COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
> COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
> COUCHDB-3289: handle error clause when calling fabric:open_revs
> COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
> COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
> COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
> COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
> COUCHDB-3364: Don’t crash compactor when compacting process fails.
> COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
> COUCHDB-3376: Fix mem3_shards under load
> COUCHDB-3378: Fix mango full text detection
> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
> COUCHDB-3402: race condition in mem3 startup
> #511: (mango) Return false for empty list
> #595: Return 409 to PUT attachment with non-existent rev
> #623: Ensure replicator _active_tasks entry reports recent pending changes value
> #627: Pass UserCtx to fabric’s all_docs from mango query
> #631: fix couchdb_os_proc_pool eunit timeouts
> #644: Make couch_event_sup:stop/1 synchronous
> #645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
> #648: Fix couch_replicator_changes_reader:process_change
> #649: Avoid a race when restarting an index updater
> #667: Prevent a terrible race condition
> #677: Make replication filter fetch error for _replicate return a 404
> Fix CORS max_age configuration parameter via Access-Control-Max-Age
> Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
> Allow w parameter for attachments
> Return “Bad Request” when count in /_uuids exceeds max
> Fix crashes when replicator db is deleted
> Skip internal replication if changes already replicated
> Fix encoding issues on _update/../doc_id and PUT attachments
>
> Upgrade Notes
>
> The deprecated (and broken) OAuth 1.0 implementation has been removed.
> If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
> The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
> The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Robert Samuel Newson <rn...@apache.org>.
Thanks to everyone for the amazing efforts exerted to get here but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
Best,
B.
> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
>
> Dear community,
>
> Apache CouchDB 2.1.0 has been released and is available for
> download.
>
> CouchDB is a database that completely embraces the web. Store your data
> with JSON documents. Access your documents with your web browser, via
> HTTP. Query, combine, and transform your documents with JavaScript.
> CouchDB works well with modern web and mobile apps. You can distribute
> your data, efficiently using CouchDB’s incremental replication. CouchDB
> supports master-master setups with automatic conflict detection.
>
> CouchDB comes with a suite of features, such as on-the-fly document
> transformation and real-time change notifications, that make web
> development a breeze. It even comes with an easy to use web
> administration console, served directly out of CouchDB! We care a lot
> about distributed scaling. CouchDB is highly available and partition
> tolerant, but is also eventually consistent. And we care a lot about
> your data. CouchDB has a fault-tolerant storage engine that puts the
> safety of your data first.
>
> Download your copy here:
>
> https://couchdb.apache.org/#download
>
> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> RHEL/CentOS are available.
>
> CouchDB 2.1.0 is a feature release, and was originally published on
> 2017-08-07.
>
> The community would like to thank all contributors for their part in
> making this release, from the smallest bug report or patch to major
> contributions in code, design, or marketing, we couldn’t have done it
> without you!
>
> The release notes follow. A formatted version of them can be viewed
> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
>
> On behalf of the CouchDB PMC,
> Joan Touzet
>
> ---
>
> Version 2.1.0
>
> The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
>
> New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
> _scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
> _scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
>
> By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
>
> Other scheduling replicator improvements
>
> Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
> Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
> Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
> Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
> Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
>
> The 2.1.0 release also includes the following minor improvements:
>
> COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
> COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
> COUCHDB-2988: (mango) Allow query selector as changes and replication filter
> COUCHDB-2992: Add additional support for document size
> COUCHDB-3046: Improve reduce function overflow protection
> COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
> COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
> COUCHDB-3180: Add features list in the welcome message
> COUCHDB-3203: Make auth handlers configurable (in ini files)
> COUCHDB-3234: Track open shard timeouts with a counter instead of logging
> COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
> COUCHDB-3249: Add config to disable index all fields (text indexes)
> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
> COUCHDB-3298: Optimize writing btree nodes
> COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
> COUCHDB-3318: bypass couch_httpd_vhost if there are none
> COUCHDB-3323: Idle dbs cause excessive overhead
> COUCHDB-3324: Introduce couch_replicator_scheduler
> COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
> COUCHDB-3358: (mango) Use efficient set storage for field names
> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
> #457: TeX/LaTeX/texinfo removed from default docs build chain
> #469: (mango) Choose index based on fields match
> #483: couchup database migration tool
> #582: Add X-Frame-Options support to help protect against clickjacking
> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
> #624: Enable compaction daemon by default
> #626: Allow enable node decom using string “true”
> (mango) Configurable default limit, defaults to 25.
> (mango) _design documents ignored when querying _all_docs
> (mango) add $allMatch selector
> Add local.d/default.d directories by default and document
> Improved INSTALL.* text files
>
> Fixed Issues
>
> The 2.1.0 release includes fixes for the following issues:
>
> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
> COUCHDB-2731: Authentication DB was not considered a system DB
> COUCHDB-3010: (Superceded fix for replication exponential backoff)
> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
> COUCHDB-3100: Fix documentation on require_valid_user
> COUCHDB-3109: 500 when include_docs=true for linked documents
> COUCHDB-3113: fabric:open_revs can return {ok, []}
> COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
> COUCHDB-3150: Update all shards with stale=update_after
> COUCHDB-3158: Fix a crash when connection closes for _update
> COUCHDB-3162: Default ssl settings cause a crash
> COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
> COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
> COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
> COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
> COUCHDB-3178: Fabric does not send message when filtering lots of documents
> COUCHDB-3181: function_clause error when adding attachment to doc in _users db
> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
> COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
> COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
> COUCHDB-3202: (mango) do not allow empty field names
> COUCHDB-3220: Handle timeout in _revs_diff
> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
> COUCHDB-3231: Allow fixing users’ documents (type and roles)
> COUCHDB-3232: user context not passed down in fabric_view_all_docs
> COUCHDB-3238: os_process_limit documentation wrong
> COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
> COUCHDB-3245: Make couchjs -S option take effect again
> COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
> COUCHDB-3255: Conflicts introduced by recreating docs with attachments
> COUCHDB-3259: Don’t trap exits in couch_file
> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> COUCHDB-3269: view response can ‘hang’ with filter and limit specified
> COUCHDB-3271: Replications crash with ‘kaboom’ exit
> COUCHDB-3274: eof in couch_file can be incorrect after error
> COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
> COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
> COUCHDB-3289: handle error clause when calling fabric:open_revs
> COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
> COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
> COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
> COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
> COUCHDB-3364: Don’t crash compactor when compacting process fails.
> COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
> COUCHDB-3376: Fix mem3_shards under load
> COUCHDB-3378: Fix mango full text detection
> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
> COUCHDB-3402: race condition in mem3 startup
> #511: (mango) Return false for empty list
> #595: Return 409 to PUT attachment with non-existent rev
> #623: Ensure replicator _active_tasks entry reports recent pending changes value
> #627: Pass UserCtx to fabric’s all_docs from mango query
> #631: fix couchdb_os_proc_pool eunit timeouts
> #644: Make couch_event_sup:stop/1 synchronous
> #645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
> #648: Fix couch_replicator_changes_reader:process_change
> #649: Avoid a race when restarting an index updater
> #667: Prevent a terrible race condition
> #677: Make replication filter fetch error for _replicate return a 404
> Fix CORS max_age configuration parameter via Access-Control-Max-Age
> Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
> Allow w parameter for attachments
> Return “Bad Request” when count in /_uuids exceeds max
> Fix crashes when replicator db is deleted
> Skip internal replication if changes already replicated
> Fix encoding issues on _update/../doc_id and PUT attachments
>
> Upgrade Notes
>
> The deprecated (and broken) OAuth 1.0 implementation has been removed.
> If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
> The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
> The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.
Re: [ANNOUNCE] Apache CouchDB 2.1.0 released
Posted by Robert Samuel Newson <rn...@apache.org>.
Thanks to everyone for the amazing efforts exerted to get here but a big and special thanks to Joan Touzet who made it her mission to get this done and succeeded.
Best,
B.
> On 7 Aug 2017, at 19:09, Joan Touzet <wo...@apache.org> wrote:
>
> Dear community,
>
> Apache CouchDB 2.1.0 has been released and is available for
> download.
>
> CouchDB is a database that completely embraces the web. Store your data
> with JSON documents. Access your documents with your web browser, via
> HTTP. Query, combine, and transform your documents with JavaScript.
> CouchDB works well with modern web and mobile apps. You can distribute
> your data, efficiently using CouchDB’s incremental replication. CouchDB
> supports master-master setups with automatic conflict detection.
>
> CouchDB comes with a suite of features, such as on-the-fly document
> transformation and real-time change notifications, that make web
> development a breeze. It even comes with an easy to use web
> administration console, served directly out of CouchDB! We care a lot
> about distributed scaling. CouchDB is highly available and partition
> tolerant, but is also eventually consistent. And we care a lot about
> your data. CouchDB has a fault-tolerant storage engine that puts the
> safety of your data first.
>
> Download your copy here:
>
> https://couchdb.apache.org/#download
>
> Pre-built convenience binaries for Windows, macOS, Debian/Ubuntu and
> RHEL/CentOS are available.
>
> CouchDB 2.1.0 is a feature release, and was originally published on
> 2017-08-07.
>
> The community would like to thank all contributors for their part in
> making this release, from the smallest bug report or patch to major
> contributions in code, design, or marketing, we couldn’t have done it
> without you!
>
> The release notes follow. A formatted version of them can be viewed
> at http://docs.couchdb.org/en/2.1.0/whatsnew/2.1.html
>
> On behalf of the CouchDB PMC,
> Joan Touzet
>
> ---
>
> Version 2.1.0
>
> The Mango _find endpoint supports a new combination operator, $allMatch, which matches and returns all documents that contain an array field with all its elements matching all the specified query criteria.
>
> New scheduling replicator. The core of the new replicator is a scheduler which allows running a large number of replication jobs by switching between them, stopping some and starting others periodically. Jobs which fail are backed off exponentially. There is also an improved inspection and querying API: _scheduler/jobs and _scheduler/docs:
> _scheduler/jobs : This endpoint shows active replication jobs. These are jobs managed by the scheduler. Some of them might be running, some might be waiting to run, or backed off (penalized) because they crashed too many times. Semantically this is somewhat equivalent to _active_tasks but focuses only on replications. Jobs which have completed or which were never created because of malformed replication documents will not be shown here as they are not managed by the scheduler. _replicate replications, started form _replicate endpoint not from a document in a _replicator db, will also show up here.
> _scheduler/docs : This endpoint is an improvement on having to go back and read replication documents to query their state. It represents the state of all the replications started from documents in _replicator db. Unlike _scheduler/jobs it will also show jobs which have failed or have completed.
>
> By default, scheduling replicator will not update documents with transient states like triggered or error anymore, instead _scheduler/docs API should be used to query replication document states.
>
> Other scheduling replicator improvements
>
> Network resource usage and performance was improved by implementing a shared connection pool. This should help in cases of a large number of connections to the same sources or target. Previously connection pools were shared only withing a single replication job.
> Improved request rate limit handling. Replicator requests will auto-discover rate limit capacity on targets and sources based on a proven Additive Increase / Multiplicative Decrease feedback control algorithm.
> Improved performance by having exponential backoff for all replication jobs failures. Previously there were some scenarios were failure led to continuous repeated retries, consuming CPU and disk resources in the process.
> Improved recovery from long but temporary network failure. Currently if replications jobs fail to start 10 times in a row, they will not be retried anymore. This is sometimes desirable, but in some cases, for example, after a sustained DNS failure which eventually recovers, replications reach their retry limit, stop retrying and never recover. Previously it required user intervention to continue. Scheduling replicator will never give up retrying a valid scheduled replication job and so it should recover automatically.
> Better handling of filtered replications. Failing user filter code fetches from the source will not block replicator manager and stall other replications. Failing filter fetches will also be backed off exponentially. Another improvement is when filter code changes on the source, a running replication will detect that and restart itself with a new replication ID automatically.
>
> The 2.1.0 release also includes the following minor improvements:
>
> COUCHDB-1946: Hibernate couch_stream after each write (up to 70% reduction in memory usage during replication of DBs with large attachments)
> COUCHDB-2964: Investigate switching replicator manager change feeds to using “normal” instead of “longpoll”
> COUCHDB-2988: (mango) Allow query selector as changes and replication filter
> COUCHDB-2992: Add additional support for document size
> COUCHDB-3046: Improve reduce function overflow protection
> COUCHDB-3061: Use vectored reads to search for buried headers in .couch files. “On a modern linux system with SSD, we see improvements up to 15x.”
> COUCHDB-3063: “stale=ok” option replaced with new “stable” and “update” options.
> COUCHDB-3180: Add features list in the welcome message
> COUCHDB-3203: Make auth handlers configurable (in ini files)
> COUCHDB-3234: Track open shard timeouts with a counter instead of logging
> COUCHDB-3242: Make get view group info timeout in couch_indexer configurable
> COUCHDB-3249: Add config to disable index all fields (text indexes)
> COUCHDB-3251: Remove hot loop usage of filename:rootname/1
> COUCHDB-3284: 8Kb read-ahead in couch_file causes extra IO and binary memory usage
> COUCHDB-3298: Optimize writing btree nodes
> COUCHDB-3302: (Improve) Attachment replication over low bandwidth network connections
> COUCHDB-3307: Limit calls to maybe_add_sys_db_callbacks to once per db open
> COUCHDB-3318: bypass couch_httpd_vhost if there are none
> COUCHDB-3323: Idle dbs cause excessive overhead
> COUCHDB-3324: Introduce couch_replicator_scheduler
> COUCHDB-3337: End-point _local_docs doesn’t conform to query params of _all_docs
> COUCHDB-3358: (mango) Use efficient set storage for field names
> COUCHDB-3425: Make _doc_ids _changes filter fast-path limit configurable
> #457: TeX/LaTeX/texinfo removed from default docs build chain
> #469: (mango) Choose index based on fields match
> #483: couchup database migration tool
> #582: Add X-Frame-Options support to help protect against clickjacking
> #593: Allow bind address of 127.0.0.1 in _cluster_setup for single nodes
> #624: Enable compaction daemon by default
> #626: Allow enable node decom using string “true”
> (mango) Configurable default limit, defaults to 25.
> (mango) _design documents ignored when querying _all_docs
> (mango) add $allMatch selector
> Add local.d/default.d directories by default and document
> Improved INSTALL.* text files
>
> Fixed Issues
>
> The 2.1.0 release includes fixes for the following issues:
>
> COUCHDB-1447: X-Couch-Update-NewRev header is missed if custom headers are specified in response of _update handler (missed in 2.0 merge)
> COUCHDB-2731: Authentication DB was not considered a system DB
> COUCHDB-3010: (Superceded fix for replication exponential backoff)
> COUCHDB-3090: Error when handling empty “Access-Control-Request-Headers” header
> COUCHDB-3100: Fix documentation on require_valid_user
> COUCHDB-3109: 500 when include_docs=true for linked documents
> COUCHDB-3113: fabric:open_revs can return {ok, []}
> COUCHDB-3149: Exception written to the log if db deleted while there is a change feed running
> COUCHDB-3150: Update all shards with stale=update_after
> COUCHDB-3158: Fix a crash when connection closes for _update
> COUCHDB-3162: Default ssl settings cause a crash
> COUCHDB-3164: Request fails when using _changes?feed=eventsource&heartbeat=30000
> COUCHDB-3168: Replicator doesn’t handle well writing documents to a target db which has a small max_document_size
> COUCHDB-3173: Views return corrupt data for text fields containing non-BMP characters
> COUCHDB-3174: max_document_size setting can by bypassed by issuing multipart/related requests
> COUCHDB-3178: Fabric does not send message when filtering lots of documents
> COUCHDB-3181: function_clause error when adding attachment to doc in _users db
> COUCHDB-3184: couch_mrview_compactor:recompact/1 does not handle errors in spawned process
> COUCHDB-3193: fabric:open_revs returns multiple results when one of the shards has stem_interactive_updates=false
> COUCHDB-3199: Replicator VDU function doesn’t acount for an already malformed document in replicator db
> COUCHDB-3202: (mango) do not allow empty field names
> COUCHDB-3220: Handle timeout in _revs_diff
> COUCHDB-3222: (Fix) HTTP code 500 instead of 400 for invalid key during document creation
> COUCHDB-3231: Allow fixing users’ documents (type and roles)
> COUCHDB-3232: user context not passed down in fabric_view_all_docs
> COUCHDB-3238: os_process_limit documentation wrong
> COUCHDB-3241: race condition in couch_server if delete msg for a db is received before open_result msg
> COUCHDB-3245: Make couchjs -S option take effect again
> COUCHDB-3252: Include main-coffee.js in release artifact (broken CoffeeScript view server)
> COUCHDB-3255: Conflicts introduced by recreating docs with attachments
> COUCHDB-3259: Don’t trap exits in couch_file
> COUCHDB-3264: POST to _all_docs does not respect conflicts=true
> COUCHDB-3269: view response can ‘hang’ with filter and limit specified
> COUCHDB-3271: Replications crash with ‘kaboom’ exit
> COUCHDB-3274: eof in couch_file can be incorrect after error
> COUCHDB-3277: Replication manager crashes when it finds _replicator db shards which are not part of a mem3 db
> COUCHDB-3286: Validation function throwing unexpected json crashes with function_clause
> COUCHDB-3289: handle error clause when calling fabric:open_revs
> COUCHDB-3291: Excessively long document IDs prevent replicator from making progress
> COUCHDB-3293: Allow limiting length of document ID (for CouchDB proper)
> COUCHDB-3305: (mango) don’t crash with invalid input to built in reducer function
> COUCHDB-3362: DELETE attachment on non-existing document creates the document, rather than returning 404
> COUCHDB-3364: Don’t crash compactor when compacting process fails.
> COUCHDB-3367: Require server admin user for db/_compact and db_view_cleanup endpoints
> COUCHDB-3376: Fix mem3_shards under load
> COUCHDB-3378: Fix mango full text detection
> COUCHDB-3379: Fix couch_auth_cache reinitialization logic
> COUCHDB-3400: Notify couch_index_processes on all shards when ddoc updated
> COUCHDB-3402: race condition in mem3 startup
> #511: (mango) Return false for empty list
> #595: Return 409 to PUT attachment with non-existent rev
> #623: Ensure replicator _active_tasks entry reports recent pending changes value
> #627: Pass UserCtx to fabric’s all_docs from mango query
> #631: fix couchdb_os_proc_pool eunit timeouts
> #644: Make couch_event_sup:stop/1 synchronous
> #645: Pass db open options to fabric_view_map for _view and _list queries on _users DB
> #648: Fix couch_replicator_changes_reader:process_change
> #649: Avoid a race when restarting an index updater
> #667: Prevent a terrible race condition
> #677: Make replication filter fetch error for _replicate return a 404
> Fix CORS max_age configuration parameter via Access-Control-Max-Age
> Chunk missing revisions before attempting to save on target (improves replication for very conflicted, very deep revision tree documents)
> Allow w parameter for attachments
> Return “Bad Request” when count in /_uuids exceeds max
> Fix crashes when replicator db is deleted
> Skip internal replication if changes already replicated
> Fix encoding issues on _update/../doc_id and PUT attachments
>
> Upgrade Notes
>
> The deprecated (and broken) OAuth 1.0 implementation has been removed.
> If user code reads or manipulates replicator document states, consider using the [replicator] update_docs = true compatibility parameter. In that case replicator will continue updating documents with transient replication states. However, that will incure a performance cost. Consider instead switching using _scheduler/docs HTTP endpoint.
> The stale parameter for views and _find has been deprecated in favour of two new parameters: stable and update. The old stale=ok behaviour is equivalent to stable=true&update=false, and the old stale=update_after behaviour is equivalent to stable=true&update=lazy. The deprecated stale parameter will be removed in CouchDB 3.0.
> The [couchdb] max_document_size configuration parameter was renamed to [httpd] max_http_request_size to reflect its actual implementation. It has been enhanced by a new [couchdb] single_max_doc_size setting. Both are documented in the shipped default.ini file.