You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2023/06/16 00:39:17 UTC
[GitHub] [couchdb] nickva opened a new pull request, #4643: Improve emitted change feed sequence after a split
nickva opened a new pull request, #4643:
URL: https://github.com/apache/couchdb/pull/4643
When we get sequence before the split, we'll fill in the missing (now split)
ranges with a special `{split, OldNodeUUid}` marker. However, when sequences are emitted in the changes API, that will make the N- prefix (SeqSum) bounce around from higher to lower numbers, while users expect those to be mostly incrementing. So take a conservative approach and assume it will be full rewind for that ramge, and use 0 for that range. This is a purely cosmetic thing, when we decode the sequence that prefix gets thrown away anyway.
Fixes a part of issue: #4640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4643: Improve emitted change feed sequence after a split
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #4643:
URL: https://github.com/apache/couchdb/pull/4643#discussion_r1232363291
##########
src/fabric/src/fabric_view_changes.erl:
##########
@@ -425,6 +425,15 @@ pack_seqs(Workers) ->
Opaque = couch_util:encodeBase64Url(?term_to_bin(SeqList, [compressed])),
?l2b([integer_to_list(SeqSum), $-, Opaque]).
+% When we get sequence before the split, we'll fill in the missing (now split)
+% ranges with a special {split, OldNodeUUid} marker. However, when sequences
+% are emitted in the changes API, that will make the N- prefix (SeqSum) bounce
+% around from higher to lower numbers, while users expect those to be mostly
+% incrementing. So take a conservative approach and assume it will be full
+% rewind for that ramge, and use 0 for that range. This is a purely cosmetic
Review Comment:
oops, good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4643: Improve emitted change feed sequence after a split
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #4643:
URL: https://github.com/apache/couchdb/pull/4643#discussion_r1232362960
##########
src/fabric/src/fabric_view_changes.erl:
##########
@@ -425,6 +425,15 @@ pack_seqs(Workers) ->
Opaque = couch_util:encodeBase64Url(?term_to_bin(SeqList, [compressed])),
?l2b([integer_to_list(SeqSum), $-, Opaque]).
+% When we get sequence before the split, we'll fill in the missing (now split)
+% ranges with a special {split, OldNodeUUid} marker. However, when sequences
+% are emitted in the changes API, that will make the N- prefix (SeqSum) bounce
+% around from higher to lower numbers, while users expect those to be mostly
+% incrementing. So take a conservative approach and assume it will be full
+% rewind for that ramge, and use 0 for that range. This is a purely cosmetic
+% thing, when we decode the sequence that prefix gets thrown away anyway.
+%
+seq({Seq, {split, <<_/binary>>}, _Node}) when is_integer(Seq) -> 0;
Review Comment:
good idea to rename it and good point about the replicator
##########
src/fabric/src/fabric_view_changes.erl:
##########
@@ -425,6 +425,15 @@ pack_seqs(Workers) ->
Opaque = couch_util:encodeBase64Url(?term_to_bin(SeqList, [compressed])),
?l2b([integer_to_list(SeqSum), $-, Opaque]).
+% When we get sequence before the split, we'll fill in the missing (now split)
+% ranges with a special {split, OldNodeUUid} marker. However, when sequences
+% are emitted in the changes API, that will make the N- prefix (SeqSum) bounce
+% around from higher to lower numbers, while users expect those to be mostly
+% incrementing. So take a conservative approach and assume it will be full
+% rewind for that ramge, and use 0 for that range. This is a purely cosmetic
Review Comment:
oops, good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #4643: Improve emitted change feed sequence after a split
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #4643:
URL: https://github.com/apache/couchdb/pull/4643#issuecomment-1593888274
Using the script from https://github.com/apache/couchdb/issues/4640 to test it as well:
```shell
#!/bin/bash
CDB_AUTH='adm:pass'
CDB_HOST='127.0.0.1'
CDB_PORT=15984
cdb () {
curl -gs -H 'Content-Type: application/json' "$(cdb-host)$1" "${@:2}"
}
cdb-host () {
echo "http://${CDB_AUTH}@${CDB_HOST}:${CDB_PORT}"
}
stop-jobs () {
cdb '/_reshard/jobs' | jq -r '.jobs[] | .id' | while read -r id ; do
cdb "/_reshard/jobs/$id" -X DELETE
done
}
split-shards () {
local db="$1"
cdb "/$db/_shards" | jq -r '.shards | keys[]' | while read -r range ; do
cdb '/_reshard/jobs' -X POST \
-d '{ "type": "split", "db": "'$db'", "range": "'$range'" }'
done
}
wait-for-q () {
local db="$1"
local q="$2"
while [[ $(cdb "/$db" | jq -r '.cluster.q') != $q ]] ; do
sleep 1
done
}
db='test-db'
cdb "/$db" -X DELETE
cdb "/$db?q=2" -X PUT
for n in {1..10} ; do
cdb "/$db/doc-$n" -X PUT -d '{ "n": '$n' }'
done
last_seq_q2="$(cdb "/$db/_changes" | jq -r '.last_seq')"
stop-jobs
split-shards "$db"
wait-for-q "$db" 4
last_seq_q4="$(cdb "/$db/_changes" | jq -r '.last_seq')"
cdb "/$db/_changes?since=$last_seq_q2" | jq
stop-jobs
split-shards "$db"
wait-for-q "$db" 8
last_seq_q8="$(cdb "/$db/_changes" | jq -r '.last_seq')"
cdb "/$db/_changes?since=$last_seq_q4" | jq
cdb "/$db/_changes?since=$last_seq_q2" | jq
```
```
./repro.sh
{"error":"not_found","reason":"Database does not exist."}
{"ok":true}
{"ok":true,"id":"doc-1","rev":"1-731bef401491606a3b246ed178e697c1"}
{"ok":true,"id":"doc-2","rev":"1-43a36d04d31d38efc5b9245c53e00627"}
{"ok":true,"id":"doc-3","rev":"1-2acff5ec499bf1cf4a478896765b824e"}
{"ok":true,"id":"doc-4","rev":"1-2371fbbf6336dcb0ff571e1f4f449ad0"}
{"ok":true,"id":"doc-5","rev":"1-16599e9b349642aef785eac10b9f94a3"}
{"ok":true,"id":"doc-6","rev":"1-cd1f7574f3318036b66bb76bd4f7ed91"}
{"ok":true,"id":"doc-7","rev":"1-f996a2834eb8c362514d171f99324abd"}
{"ok":true,"id":"doc-8","rev":"1-817027c9296c2d2d688c568237e89624"}
{"ok":true,"id":"doc-9","rev":"1-46d26f2360296f940c344ef881b52066"}
{"ok":true,"id":"doc-10","rev":"1-6829757f6498d33021e0a764f0a492af"}
[{"ok":true,"id":"001-486d0ee2a74434c94bec8f9a7034842bd5882af99ea9edee32c9f8222ff916e8","node":"node1@127.0.0.1","shard":"shards/00000000-7fffffff/test-db.1686875897"}]
[{"ok":true,"id":"001-bafee5292431febf513fedab86c793bf770513576f812b3f032d55aca0f1a83d","node":"node1@127.0.0.1","shard":"shards/80000000-ffffffff/test-db.1686875897"}]
{
"results": [],
"last_seq": "20-g1AAAAEjeJzLYWBgYMlgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_f___7MymBNZcoEC7IbG5gZmhmaYynEakeQAJJPqUUxJSzY0tUgxJsGUPBYgydAApIAG7QeZxAY2ydjCOMUkKY1kkw5ATPqPMMk8JdnEMjEVU1cWACVXR8c",
"pending": 0
}
{"ok":true}
{"ok":true}
[{"ok":true,"id":"001-c48499ca13a5a9602e6bf1a23c9f38744f71fa0bf9c4d949de3414f90e69b82f","node":"node1@127.0.0.1","shard":"shards/00000000-3fffffff/test-db.1686875897"}]
[{"ok":true,"id":"001-a6b3d236b0e1e865dcd90fed71de21871828ebfa080ca9e80b3891f96a302515","node":"node1@127.0.0.1","shard":"shards/40000000-7fffffff/test-db.1686875897"}]
[{"ok":true,"id":"001-d0c624fb555c48d3aa45b99750d5093703b9ea31ac3d69701995ccf9b70c9945","node":"node1@127.0.0.1","shard":"shards/80000000-bfffffff/test-db.1686875897"}]
[{"ok":true,"id":"001-6e67e89cb8171afd46d59b9208ee69d2c8f80085ae320d1ced00e30139bec65e","node":"node1@127.0.0.1","shard":"shards/c0000000-ffffffff/test-db.1686875897"}]
{ "results": [],
"last_seq": "34-g1AAAAJDeJyV0FEKwjAMANDihvrnEfQE0s5lXb_cTbRZK2PMeQS9id5Eb6I3qZkV5hh-lEACafNo2jDG5lVk2KI9GSsKkcg1pxANHUw0w6Vzrq4iHR-pMUuMzfgBxtf_EriijNuBYkACbLIQpeiU3VeJPkoq-SYXZYiy75TzQJHaKmVD3tLGlNmFCkHXfivgkKMMl25euveS1oh5kgZLDy89O2nq_yhVtgQVLL285H4kQK6sGE_Vb5jejjE",
"pending": 0
}
{ "results": [
{
"seq": "2-g1AAAAKJeJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBOZcoEC7EYpqWYGaaaYynEakaQAJJPsoaawZDClMLAWF-RkloDNM0k0NjK0TCXFPAeQefHUMy8BZF491czLYwGSDA1ACmjkfJCZbGhmJpqbJVumWZJs5gKImfupaeYBiJn3qWnmA4iZ_0kyMwsA0VGkAA",
"id": "doc-3",
"changes": [
{
"rev": "1-2acff5ec499bf1cf4a478896765b824e"
}
]
},
{
"seq": "3-g1AAAAJ_eJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBOZcoEC7EYpqWYGaaaYynEakaQAJJPsoaawZDClMLAWF-RkloDNM0k0NjK0TCXFPAeQefHUMy8BZF491czLYwGSDA1ACmjkfJCZbGhmJpqbJVumWZJs5gKImfupaeYBiJn3qWnmA4iZ4PBkhISiaZKBZaohpq4sAF0RoL0",
"id": "doc-1",
"changes": [
{
"rev": "1-731bef401491606a3b246ed178e697c1"
}
]
},
{
"seq": "5-g1AAAAJreJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBNZcoEC7EYpqWYGaaaYynEakaQAJJPsoaYwgE1JMTU3NTU2I8UUB5Ap8SimmJgbGFsYJpNiSgLIlHqYjzKYUhhYiwtyMksg5iUaGxlappJgXh4LkGRoAFJAI-eDzGRDMzPR3CzZMs2SZDMXQMzcT00zD0DMvE9NMx9AzASHJyMkFE2TDCxTDTF1ZQEAnFiaZA",
"id": "doc-7",
"changes": [
{
"rev": "1-f996a2834eb8c362514d171f99324abd"
}
]
},
{
"seq": "7-g1AAAAJreJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBNZcoEC7EYpqWYGaaaYynEakaQAJJPsoaYwgE1JMTU3NTU2I8UUB5Ap8SimmJgbGFsYJpNiSgLIlHqYjzKYUhhYiwtyMksg5iUaGxlappJgXh4LkGRoAFJAI-eDzGRDMzPR3CzZMs2SZDMXQMzcT00zD0DMvE9NMx9AzASHJzMkFE2TDCxTDTF1ZQEAnJiaZg",
"id": "doc-5",
"changes": [
{
"rev": "1-16599e9b349642aef785eac10b9f94a3"
}
]
},
{
"seq": "8-g1AAAAJheJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBNZcoEC7EYpqWYGaaaYynEakaQAJJPsoaYwgE1JMTU3NTU2I8UUB5Ap8SimmJgbGFsYJpNiSgLIlHqoKYxgU8wTUy0tU0lxSx4LkGRoAFJAg-aDTGLLYEphYC0uyMksAZuZaG6WbJlmSbKZCyBm7qemmQcgZt6nppkPIGaCQ5EZEhemSQaWqYaYurIASVGXpQ",
"id": "doc-2",
"changes": [
{
"rev": "1-43a36d04d31d38efc5b9245c53e00627"
}
]
},
{
"seq": "10-g1AAAAJheJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBNZcoEC7EYpqWYGaaaYynEakaQAJJPsoaYwgE1JMTU3NTU2I8UUB5Ap8SimmJgbGFsYJpNiSgLIlHqoKYxgU8wTUy0tU0lxSx4LkGRoAFJAg-aDTGLLYEphYC0uyMksAZuZaG6WbJlmSbKZCyBm7qemmQcgZt6nppkPIGaCQ5EVEhemSQaWqYaYurIASZGXpw",
"id": "doc-9",
"changes": [
{
"rev": "1-46d26f2360296f940c344ef881b52066"
}
]
},
{
"seq": "12-g1AAAAJheJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBNZcoEC7EYpqWYGaaaYynEakaQAJJPsoaYwgE1JMTU3NTU2I8UUB5Ap8SimmJgbGFsYJpNiSgLIlHqoKcxgU8wTUy0tU0lxSx4LkGRoAFJAg-aDTGLLYEphYC0uyMksAZuZaG6WbJlmSbKZCyBm7qemmQcgZt6nppkPIGaCQ5EVEhemSQaWqYaYurIATF2XqQ",
"id": "doc-6",
"changes": [
{
"rev": "1-cd1f7574f3318036b66bb76bd4f7ed91"
}
]
},
{
"seq": "18-g1AAAAJNeJyV0EsOwiAQBuBJH-rOI-gJDLSllJW9iUKhqU2tJrrXm-hN9CZ6k0qhiTGNC0IyJAN8-ZkGAGaVL2HeHqTCOY7oCumFG33gcRCLruvqyufBXjemkVQpKsn4-l9CLHUV60EBo0hCCYlTFyXvlc2PklAUZ7hwUba9chkU3yiUK8aUS5Y20BWuetPQrZcmlSchPB2b3dmYnKYFK5mzebfm4_tHzoXIosRZelrpZdLZaSVMFcQ909tKZmKhlYhATOHxq_oDFP6Rjg",
"id": "doc-10",
"changes": [
{
"rev": "1-6829757f6498d33021e0a764f0a492af"
}
]
},
{
"seq": "20-g1AAAAJDeJyV0N0NgjAQAOAL4M-bI-gEpgWO0ifZRFtaQwjiCLqJbqKb6CbYHxJjiA_NJXfJtf3Suw4Alk2sYNWflaYVTdmWmKCdOYgEyPUwDG0Ti-RkGotU6YIccXr9LyE3JsvdqIBTFDLErAhRKqvsf5SckaykdYhysMplVGKnMKE51yF_6ROT4WqKgW5WipyEBEvJwqW7lx7fyYSQskzzYOnppZeV5n5HOdc18mDp7SW3p5mXUBKu6fRV-wGH-I4j",
"id": "doc-4",
"changes": [
{
"rev": "1-2371fbbf6336dcb0ff571e1f4f449ad0"
}
]
},
{
"seq": "22-g1AAAAJDeJyV0N0NgjAQB_CL4MebI-gEpgWO0ifZRHu0hhDEEXQT3UQ30U2wUBJiiA9Nk2vSa3-5_msAWJWBhnVz0YbnPBI7ZhevbWOmgDZt21ZloMKzPVhG2qTshNPrfwna2kr7QYFe0SgQ49RHyTvl8KMkgsUZL3yUY6dcByXoFaGMlMZnlia0FW52s9B9zAYZZiT8pYeTnuPPlCLKosRbejnp3UkLl1EiTYHSW_o4qc9p7iQkJg2fvqq-ifSOJQ",
"id": "doc-8",
"changes": [
{
"rev": "1-817027c9296c2d2d688c568237e89624"
}
]
}
],
"last_seq": "22-g1AAAAJDeJyV0N0NgjAQB_CL4MebI-gEpgWO0ifZRHu0hhDEEXQT3UQ30U2wUBJiiA9Nk2vSa3-5_msAWJWBhnVz0YbnPBI7ZhevbWOmgDZt21ZloMKzPVhG2qTshNPrfwna2kr7QYFe0SgQ49RHyTvl8KMkgsUZL3yUY6dcByXoFaGMlMZnlia0FW52s9B9zAYZZiT8pYeTnuPPlCLKosRbejnp3UkLl1EiTYHSW_o4qc9p7iQkJg2fvqq-ifSOJQ",
"pending": 0
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] pgj commented on a diff in pull request #4643: Improve emitted change feed sequence after a split
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4643:
URL: https://github.com/apache/couchdb/pull/4643#discussion_r1232017156
##########
src/fabric/src/fabric_view_changes.erl:
##########
@@ -425,6 +425,15 @@ pack_seqs(Workers) ->
Opaque = couch_util:encodeBase64Url(?term_to_bin(SeqList, [compressed])),
?l2b([integer_to_list(SeqSum), $-, Opaque]).
+% When we get sequence before the split, we'll fill in the missing (now split)
+% ranges with a special {split, OldNodeUUid} marker. However, when sequences
+% are emitted in the changes API, that will make the N- prefix (SeqSum) bounce
+% around from higher to lower numbers, while users expect those to be mostly
+% incrementing. So take a conservative approach and assume it will be full
+% rewind for that ramge, and use 0 for that range. This is a purely cosmetic
Review Comment:
s/ramge/range/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4643: Improve emitted change feed sequence after a split
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #4643:
URL: https://github.com/apache/couchdb/pull/4643#discussion_r1232407816
##########
src/fabric/src/fabric_view_changes.erl:
##########
@@ -425,6 +425,15 @@ pack_seqs(Workers) ->
Opaque = couch_util:encodeBase64Url(?term_to_bin(SeqList, [compressed])),
?l2b([integer_to_list(SeqSum), $-, Opaque]).
+% When we get sequence before the split, we'll fill in the missing (now split)
+% ranges with a special {split, OldNodeUUid} marker. However, when sequences
+% are emitted in the changes API, that will make the N- prefix (SeqSum) bounce
+% around from higher to lower numbers, while users expect those to be mostly
+% incrementing. So take a conservative approach and assume it will be full
+% rewind for that ramge, and use 0 for that range. This is a purely cosmetic
+% thing, when we decode the sequence that prefix gets thrown away anyway.
+%
+seq({Seq, {split, <<_/binary>>}, _Node}) when is_integer(Seq) -> 0;
Review Comment:
renamed the function name and updated the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva merged pull request #4643: Improve emitted change feed sequence after a split
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva merged PR #4643:
URL: https://github.com/apache/couchdb/pull/4643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rnewson commented on a diff in pull request #4643: Improve emitted change feed sequence after a split
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #4643:
URL: https://github.com/apache/couchdb/pull/4643#discussion_r1231928309
##########
src/fabric/src/fabric_view_changes.erl:
##########
@@ -425,6 +425,15 @@ pack_seqs(Workers) ->
Opaque = couch_util:encodeBase64Url(?term_to_bin(SeqList, [compressed])),
?l2b([integer_to_list(SeqSum), $-, Opaque]).
+% When we get sequence before the split, we'll fill in the missing (now split)
+% ranges with a special {split, OldNodeUUid} marker. However, when sequences
+% are emitted in the changes API, that will make the N- prefix (SeqSum) bounce
+% around from higher to lower numbers, while users expect those to be mostly
+% incrementing. So take a conservative approach and assume it will be full
+% rewind for that ramge, and use 0 for that range. This is a purely cosmetic
+% thing, when we decode the sequence that prefix gets thrown away anyway.
+%
+seq({Seq, {split, <<_/binary>>}, _Node}) when is_integer(Seq) -> 0;
Review Comment:
it's slightly more than cosmetic, the replicator (or at least older versions) depending on an increasing term in order to checkpoint regularly. If that's no longer the case we should consider dropping this otherwise useless number.
I think the change here is fine and the comment will help us in future, however I would like us to rename the `seq/1` method to more clearly express that its for the fake packed_seq prefix. as currently named it implies it extracts the sequence from various tuple formats (which of course it _did_ before this change). Either we rename it to something like `fake_packed_seq/1` or leave it as is and do the conversion to 0 for split tuples in another function with an explanatory name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org