You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by va...@apache.org on 2026/04/29 13:53:31 UTC
(couchdb) branch main updated (7039e0be8 -> c82fe8ffb)
This is an automated email from the ASF dual-hosted git repository.
nickva pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
from 7039e0be8 Merge pull request #5984 from apache/disable-copilot-code-review
new e979f7f8b Do not run full GC after each doc update in the indexer
new 2c4b6a138 Increase view queue size from 100KB to 10MB
new d7aefe0f7 Use config:get_integer/3 in couch_mrview_updater
new c82fe8ffb Make view compaction tests unflaky
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
rel/overlay/etc/default.ini | 15 +++++++++++---
.../test/eunit/couchdb_file_compression_tests.erl | 8 ++++++++
src/couch_index/src/couch_index_updater.erl | 23 +++++++++++++++-------
src/couch_mrview/src/couch_mrview_updater.erl | 9 ++++-----
4 files changed, 40 insertions(+), 15 deletions(-)
(couchdb) 01/04: Do not run full GC after each doc update in the indexer
Posted by va...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nickva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit e979f7f8b448043b95f68214faf73790e4d2cfd1
Author: Nick Vatamaniuc <va...@gmail.com>
AuthorDate: Wed Apr 29 00:25:30 2026 -0400
Do not run full GC after each doc update in the indexer
Run it after 1000 docs, ensuring it still forced to run as a safe default.
However, in OTP 27+ this should not be needed at all. So infinity is also a
possible config setting, we may change to that in the future.
Comments from John Hogberg in OTP repo regarind behavior change in OTP 27:
https://github.com/erlang/otp/issues/8229#issuecomment-1988858134
> the GC pressure of off-heap binaries ("vheap") was vastly under-counted prior
to 24ef4cb [1]. In these tests this caused it to GC less often and, crucially, when
there was less live data to keep.
[1]
https://github.com/erlang/otp/commit/24ef4cbaeda9b9c26682cba75f2f15b0c58722aa
---
rel/overlay/etc/default.ini | 9 ++++++++-
src/couch_index/src/couch_index_updater.erl | 23 ++++++++++++++++-------
2 files changed, 24 insertions(+), 8 deletions(-)
diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini
index bb017fb44..df24e1d44 100644
--- a/rel/overlay/etc/default.ini
+++ b/rel/overlay/etc/default.ini
@@ -491,7 +491,7 @@ hash_algorithms = sha256, sha
;max_write_delay = 500
;update_db = true
-;[view_updater]
+[view_updater]
; Configure the queue capacity used during indexing. These settings apply to
; both the queue between the changes feed and the JS mapper, and between the
; JS mapper and the disk writer.
@@ -505,6 +505,13 @@ hash_algorithms = sha256, sha
;min_writer_items = 100
;min_writer_size = 16777216
+; After how many processed docs to run garbage collection in view index updater
+; process (infinity is a also a possible setting, to let the Erlang VM run GC
+; as it sees fit), version >= 27 should do that much better than before
+; Previously in versions < 3.5.1 this this was running after every single doc
+; update
+;gc_interval_docs = 1000
+
[couch_httpd_auth]
; WARNING! This only affects the node-local port (5986 by default).
; You probably want the settings under [chttpd].
diff --git a/src/couch_index/src/couch_index_updater.erl b/src/couch_index/src/couch_index_updater.erl
index c7596e90d..3ae18750a 100644
--- a/src/couch_index/src/couch_index_updater.erl
+++ b/src/couch_index/src/couch_index_updater.erl
@@ -159,15 +159,24 @@ update(Idx, Mod, IdxState) ->
end
end,
- Proc = fun(DocInfo, {IdxStateAcc, _}) ->
- case CommittedOnly and (GetSeq(DocInfo) > DbCommittedSeq) of
+ GcInterval = config:get_integer_or_infinity("view_updater", "gc_interval_docs", 1000),
+ Proc = fun(DocInfo, {IdxStateAcc, _, NDocs}) ->
+ case CommittedOnly andalso (GetSeq(DocInfo) > DbCommittedSeq) of
true ->
- {stop, {IdxStateAcc, false}};
+ {stop, {IdxStateAcc, false, NDocs}};
false ->
{Doc, Seq} = LoadDoc(DocInfo),
{ok, NewSt} = Mod:process_doc(Doc, Seq, IdxStateAcc),
- garbage_collect(),
- {ok, {NewSt, true}}
+ NDocs1 = NDocs + 1,
+ case GcInterval of
+ infinity ->
+ ok;
+ _ when NDocs1 rem GcInterval == 0 ->
+ garbage_collect();
+ _ ->
+ ok
+ end,
+ {ok, {NewSt, true, NDocs1}}
end
end,
{ok, InitIdxState} = Mod:start_update(
@@ -177,9 +186,9 @@ update(Idx, Mod, IdxState) ->
NumPurgeChanges
),
- Acc0 = {InitIdxState, true},
+ Acc0 = {InitIdxState, true, 0},
{ok, Acc} = couch_db:fold_changes(Db, CurrSeq, Proc, Acc0, []),
- {ProcIdxSt, SendLast} = Acc,
+ {ProcIdxSt, SendLast, _} = Acc,
% If we didn't bail due to hitting the last committed seq we need
% to send our last update_seq through.
(couchdb) 04/04: Make view compaction tests unflaky
Posted by va...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nickva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit c82fe8ffb3e5169d78be4cae6696ed0a3908b935
Author: Nick Vatamaniuc <va...@gmail.com>
AuthorDate: Wed Apr 29 04:14:10 2026 -0400
Make view compaction tests unflaky
It turns after we made view updates more efficient, some view compaction tests
started to flake out because they relied on view builds batching inneficiently
always and creating garbage between the batches.
To fix it, do not rely on inneficient batching, instead just do another
explicit fresh and update, this should guarantee at least some extra garbage
being created in the tree.
---
src/couch/test/eunit/couchdb_file_compression_tests.erl | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/src/couch/test/eunit/couchdb_file_compression_tests.erl b/src/couch/test/eunit/couchdb_file_compression_tests.erl
index 1016cd2de..c3ec4174a 100644
--- a/src/couch/test/eunit/couchdb_file_compression_tests.erl
+++ b/src/couch/test/eunit/couchdb_file_compression_tests.erl
@@ -70,6 +70,14 @@ setup() ->
{ok, _} = couch_db:update_doc(Db, DDoc, []),
ok = couch_db:close(Db),
ok = refresh_index(DbName),
+ % Another update and refresh. A forced second b-tree update will guarantee
+ % to create garbage so compaction has something to chew on. Otherwise a
+ % single bulk build can pack the tree optimally so compact_view won't have
+ % anything to do.
+ {ok, Db2} = couch_db:open_int(DbName, [?ADMIN_CTX]),
+ ok = populate_db(Db2, ?DOCS_COUNT),
+ ok = couch_db:close(Db2),
+ ok = refresh_index(DbName),
DbName.
teardown(DbName) ->
(couchdb) 02/04: Increase view queue size from 100KB to 10MB
Posted by va...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nickva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 2c4b6a138b981105b293d0e375adca1c72ec24b0
Author: Nick Vatamaniuc <va...@gmail.com>
AuthorDate: Wed Apr 29 00:43:42 2026 -0400
Increase view queue size from 100KB to 10MB
100KB was undersized given the 100 batch size used in the process_doc call.
100 docs as small as 1KB will already start to get serialized and not take
advantage of the work queue buffering. Make it 10MB to fit about 10 batches of
1MB each and be a bit under the min_writer_size default of 16MB.
---
rel/overlay/etc/default.ini | 6 ++++--
src/couch_mrview/src/couch_mrview_updater.erl | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini
index df24e1d44..eabfc3c9b 100644
--- a/rel/overlay/etc/default.ini
+++ b/rel/overlay/etc/default.ini
@@ -497,8 +497,10 @@ hash_algorithms = sha256, sha
; JS mapper and the disk writer.
; Whichever limit happens to be hit first is the one that takes effect.
-; The maximum queue memory size
-;queue_memory_cap = 100000
+; The maximum queue memory size. view update gen_server batch size is 100, so the
+; 10MB default would keep about 10 batches of 10KB docs in the queue. Also 10MB is
+; also slightly smaller than the min_writer_size of 16MB
+;queue_memory_cap = 10485760
; The maximum queue length
;queue_item_cap = 500
diff --git a/src/couch_mrview/src/couch_mrview_updater.erl b/src/couch_mrview/src/couch_mrview_updater.erl
index 4238e6b7d..34631442c 100644
--- a/src/couch_mrview/src/couch_mrview_updater.erl
+++ b/src/couch_mrview/src/couch_mrview_updater.erl
@@ -20,7 +20,7 @@
-define(REM_VAL, removed).
start_update(Partial, State, NumChanges, NumChangesDone) ->
- MaxSize = config:get_integer("view_updater", "queue_memory_cap", 100000),
+ MaxSize = config:get_integer("view_updater", "queue_memory_cap", 10485760),
MaxItems = config:get_integer("view_updater", "queue_item_cap", 500),
QueueOpts = [{max_size, MaxSize}, {max_items, MaxItems}],
{ok, DocQueue} = couch_work_queue:new(QueueOpts),
(couchdb) 03/04: Use config:get_integer/3 in couch_mrview_updater
Posted by va...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nickva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit d7aefe0f7e599051cad25b8cb330709d264c457f
Author: Nick Vatamaniuc <va...@gmail.com>
AuthorDate: Wed Apr 29 01:55:15 2026 -0400
Use config:get_integer/3 in couch_mrview_updater
This avoids re-parsing the string values into integers when defaults are used
every time we run the check.
---
src/couch_mrview/src/couch_mrview_updater.erl | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/src/couch_mrview/src/couch_mrview_updater.erl b/src/couch_mrview/src/couch_mrview_updater.erl
index 34631442c..d642ceb5c 100644
--- a/src/couch_mrview/src/couch_mrview_updater.erl
+++ b/src/couch_mrview/src/couch_mrview_updater.erl
@@ -229,11 +229,10 @@ accumulate_writes(State, W, Acc0) ->
accumulate_more(NumDocIds, Acc) ->
% check if we have enough items now
- MinItems = config:get("view_updater", "min_writer_items", "100"),
- MinSize = config:get("view_updater", "min_writer_size", "16777216"),
+ MinItems = config:get_integer("view_updater", "min_writer_items", 100),
+ MinSize = config:get_integer("view_updater", "min_writer_size", 16777216),
CurrMem = ?term_size(Acc),
- NumDocIds < list_to_integer(MinItems) andalso
- CurrMem < list_to_integer(MinSize).
+ NumDocIds < MinItems andalso CurrMem < MinSize.
merge_results([], SeqAcc, ViewKVs, DocIdKeys) ->
{SeqAcc, ViewKVs, DocIdKeys};