You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "greatsharp (via GitHub)" <gi...@apache.org> on 2025/07/26 13:44:13 UTC
[PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
greatsharp opened a new pull request, #3077:
URL: https://github.com/apache/kvrocks/pull/3077
Why does OOM happened when doing migration?
- If slot only has string or other no-subkey data, then batch_sender will put raw-key data continuously, it makes rep_ string in WriteBatch class append string continuously, then OOM at last.
Why should we sleep for a while when function 'SockReadLine' ?
- If we send a big data by APPLYBATCH command to destination node, the destination node could not process this command rapidly, it will report 'Resource temporarily unavailable' error if we read response immediately.
- So we should sleep for a while then retry reading from fd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2247129967
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
How about add a new function like below?
`
StatusOr<std::string> SockReadLineWithRetry(int fd, int retry_times, int retry_interval_ms) {
if (retry_times <= 0 || retry_interval_ms <= 0) {
return SockReadLine(fd);
}
UniqueEvbuf evbuf;
while (--retry_times > 0) {
int ret = evbuffer_read(evbuf.get(), fd, -1);
if (ret > 0) {
break;
}
if (ret == 0) {
return Status::FromErrno("read response err: connection closed");
}
if (errno == EAGAIN || errno == EWOULDBLOCK) {
// Resource temporarily unavailable, sleep for a while
std::this_thread::sleep_for(std::chrono::milliseconds(retry_interval_ms));
continue;
}
return Status::FromErrno("read response err");
}
UniqueEvbufReadln line(evbuf.get(), EVBUFFER_EOL_CRLF_STRICT);
if (!line) {
return Status::FromErrno("read response err(empty)");
}
return std::string(line.get(), line.length);
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2251191992
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
After reviewing the raw-key-value migration process, the source node needs to send WriteBatch data sequentially, which is more suitable for blocking I/O
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2366336789
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
I was using `raw-key-value` and I got:
```
[2025-09-19T19:02:16.393369+00:00][E][slot_migrate.cc:195] [migrate] Failed to start migrating slot(s) 100. Error: failed to set import status on destination node: failed to check the response from the destination node: failed to read response: Resource temporarily unavailable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#issuecomment-3150247953
Maybe you can just keep the "fix migration oom" part in this PR, so we can merge it first.
And for the nonblocking io part we can take some time to see if we have good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2361902347
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
> I am confused here. I think the fd here is made by `util:SockConnect`. From the code, it does looks like `SockConnect` returns a blocking fd. Am I missing anything?
agree with you, so we just used the util::SockReadLineWithRetry(fd, 10, 500) to resolve this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2247141652
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
Add a new function can tell callers there is a 'Resource temporarily unavailable' error, callers can pass retry times and interval, how do you see? @git-hulk @torwig @PragmaTwice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2362125363
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
@git-hulk @PragmaTwice @zhixinwen
I still used util::SockReadLineWithRetry(fd, 10, 500) to resolve this 'Resource temporarily unavailable' error, it works well in big data cluster migration, but how could a blocking socket report EAGAIN or EWOULDBLOCK error?
Could you please give me some guide?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2366336789
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
I was using that one and I got:
```
[2025-09-19T19:02:16.393369+00:00][E][slot_migrate.cc:195] [migrate] Failed to start migrating slot(s) 100. Error: failed to set import status on destination node: failed to check the response from the destination node: failed to read response: Resource temporarily unavailable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2246930894
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
Agree with this. Perhaps you could also provide another sock read function with limited retry count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2247254359
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
Sounds great to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2363953568
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
My case is a bit different, I get this error in `checkMultipleResponses` and I think that may be due to the 1s timeout setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2251006591
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
In non-blocking IO, it's quite normal to receive a EAGAIN or EWOULDBLOCK to indicate that currently there's no data to retrieve from the stream.
In such case, if we just make it a busy retry loop (or with a sleep), we are actually converting it to a blocking socket.
And IMHO it seems not a good method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2249926518
##########
src/common/io_util.cc:
##########
@@ -322,6 +322,39 @@ StatusOr<std::string> SockReadLine(int fd) {
return std::string(line.get(), line.length);
}
+StatusOr<std::string> SockReadLineWithRetry(int fd, int retry_times, int retry_interval_ms) {
+ if (retry_times <= 0 || retry_interval_ms <= 0) {
+ return SockReadLine(fd);
+ }
+
+ UniqueEvbuf evbuf;
+ while (retry_times-- > 0) {
+ int ret = evbuffer_read(evbuf.get(), fd, -1);
+ if (ret > 0) {
+ break;
+ }
+
+ if (ret == 0) {
+ return Status::FromErrno("read response err: connection closed");
Review Comment:
```suggestion
return {Status::NotOK, "read response err: connection closed"};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2246930894
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
Agree with this. Perhaps you could also provide another sock read function with retry behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2234037157
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
Should we retry on a higher level rather than `SockReadLine`? cc @git-hulk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#issuecomment-3124285999
## [](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3077) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=3077&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=3077&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=3077&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [40.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3077&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3077&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3077)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "torwig (via GitHub)" <gi...@apache.org>.
torwig commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2232968948
##########
src/common/io_util.cc:
##########
@@ -310,8 +310,21 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
- return Status::FromErrno("read response err");
+ while (true) {
+ int ret = evbuffer_read(evbuf.get(), fd, -1);
+ if (ret > 0) {
+ break;
+ } else if (ret == 0) {
Review Comment:
Here we can make code more "flat":
```
if (ret > 0) {
break;
}
if (ret == 0) {
return Status::FromErrno("read response err: connection closed");
}
if (errno == EAGAIN || errno == EWOULDBLOCK) {
// Resource temporarily unavailable, sleep for a while
std::this_thread::sleep_for(std::chrono::milliseconds(200));
continue;
}
return Status::FromErrno("read response err");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2247129967
##########
src/common/io_util.cc:
##########
@@ -310,7 +310,22 @@ Status SockSetBlocking(int fd, int blocking) {
StatusOr<std::string> SockReadLine(int fd) {
UniqueEvbuf evbuf;
- if (evbuffer_read(evbuf.get(), fd, -1) <= 0) {
+ while (true) {
Review Comment:
How about add a new function like below?
```
StatusOr<std::string> SockReadLineWithRetry(int fd, int retry_times, int retry_interval_ms) {
if (retry_times <= 0 || retry_interval_ms <= 0) {
return SockReadLine(fd);
}
UniqueEvbuf evbuf;
while (retry_times-- > 0) {
int ret = evbuffer_read(evbuf.get(), fd, -1);
if (ret > 0) {
break;
}
if (ret == 0) {
return Status::FromErrno("read response err: connection closed");
}
if (errno == EAGAIN || errno == EWOULDBLOCK) {
// Resource temporarily unavailable, sleep for a while
std::this_thread::sleep_for(std::chrono::milliseconds(retry_interval_ms));
continue;
}
return Status::FromErrno("read response err");
}
UniqueEvbufReadln line(evbuf.get(), EVBUFFER_EOL_CRLF_STRICT);
if (!line) {
return Status::FromErrno("read response err(empty)");
}
return std::string(line.get(), line.length);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2377782790
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
I was having compaction stop due to migration and my theory is the blocked rocksdb call blocks the event loop and paused all command processing. In fact, even PING was frozen on the target node during the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp closed pull request #3077: fix(migration): fix migration oom and 'Resource temporarily unavailable' error
URL: https://github.com/apache/kvrocks/pull/3077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2379954778
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
Added some log for debugging:
```
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
// INSERT_YOUR_CODE
// Log the SO_RCVTIMEO (receive timeout) for fd
struct timeval tv;
socklen_t tv_len = sizeof(tv);
if (getsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, &tv_len) == 0) {
LOG(INFO) << "[migrate] fd " << fd << " SO_RCVTIMEO: " << tv.tv_sec << "s " << tv.tv_usec << "us";
} else {
LOG(WARNING) << "[migrate] Failed to get SO_RCVTIMEO for fd " << fd << ": " << strerror(errno);
}
std::string line = GET_OR_RET(util::SockReadLine(fd));
```
and I get `[2025-09-25T18:08:58.993539+00:00][I][batch_sender.cc:107] [migrate] fd 5488 SO_RCVTIMEO: 1s 0us`.
The 1s timeout is set because it was set in`checkMultipleResponses` which later affects `APPLYBATCH`. The `Resource temporarily unavailable` error is due to the timeout.
I think fixing compaction is the best way to go, but we should think about how to retry failure in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2363953568
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
My case is a bit different, I get this error in `checkMultipleResponses` and I think that may be due to the 1s timeout setting.
What is the full error message you see? I think it can come from several places.. Also I am not sure if evbuffer should be used on a blocking connection, that lib is intended for non-blocking fd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2362114120
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
I am also confused here, util::SocketConnect returns a blocking socket fd, but how could a blocking socket report EAGAIN or EWOULDBLOCK error?
Add below test logic after dst_fd_.Reset(*result) in SlotMigrator::startMigration() method,
`
int flags = fcntl(*result, F_GETFL, 0);
if ((flags & O_NONBLOCK) == 0) {
LOG(ERROR) << "block socket";
} else {
LOG(ERROR) << "non_block socket";
}
`
the log print out "block socket"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2387345831
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
I checked the socket SO_RCVTIMEO with your code, it is set to 1s after invoked checkMultipleResponses method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2366003014
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
> My case is a bit different, I get this error in `checkMultipleResponses` and I think that may be due to the 1s timeout setting.
>
> What is the full error message you see? I think it can come from several places.. Also I am not sure if evbuffer should be used on a blocking connection, that lib is intended for non-blocking fd.
please use raw-key-value migrate type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "zhixinwen (via GitHub)" <gi...@apache.org>.
zhixinwen commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2361832442
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
I am confused here. I think the fd here is made by `util:SockConnect`. From the code, it does looks like `SockConnect` returns a blocking fd. Am I missing anything?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "greatsharp (via GitHub)" <gi...@apache.org>.
greatsharp commented on code in PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#discussion_r2361902347
##########
src/cluster/batch_sender.cc:
##########
@@ -100,7 +100,7 @@ Status BatchSender::sendApplyBatchCmd(int fd, const rocksdb::WriteBatch &write_b
GET_OR_RET(util::SockSend(fd, redis::ArrayOfBulkStrings({"APPLYBATCH", write_batch.Data()})));
- std::string line = GET_OR_RET(util::SockReadLine(fd));
+ std::string line = GET_OR_RET(util::SockReadLineWithRetry(fd, 10, 500));
Review Comment:
> I am confused here. I think the fd here is made by `util:SockConnect`. From the code, it does looks like `SockConnect` returns a blocking fd. Am I missing anything?
agree with you, so we just used the util::SockReadLineWithRetry(fd, 10, 500) to resolve this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#issuecomment-3142208747
@greatsharp Nice shoot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(migration): fix migration oom and 'Resource temporarily unavailable' error [kvrocks]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3077:
URL: https://github.com/apache/kvrocks/pull/3077#issuecomment-3148176191
## [](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3077) **Quality Gate failed**
Failed conditions
 [44.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3077&metric=new_coverage&view=list) (required ≥ 50%)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3077)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org