You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by "DevinLeamy (via GitHub)" <gi...@apache.org> on 2024/04/24 15:26:21 UTC
[PR] [cgroups2] Update destroy to be async more robust. [mesos]
DevinLeamy opened a new pull request, #571:
URL: https://github.com/apache/mesos/pull/571
We were running into an inconsistent issue with `cgroups2::destroy`.
`cgroups2::destroy` would fail with EBUSY when removing cgroups with `rmdir`.
The error was being caused because some processes had not been killed
when `rmdir` was called on their cgroup; a cgroup with processes
cannot be destroyed.
After signalling a kill (by writing "1" to 'cgroup.kill') sometimes
processes were staying alive long enough to cause `rmdir` to fail.
Hence, we update `cgroups2::destroy` to wait after signalling a
SIGKILL for all the processes to drop before attempting to remove
the cgroups.
Since we wait a maximum of half a second, we don't want to block the
caller. Thus, we update `destroy` to be async.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cgroups2] Update destroy to be async more robust. [mesos]
Posted by "DevinLeamy (via GitHub)" <gi...@apache.org>.
DevinLeamy commented on code in PR #571:
URL: https://github.com/apache/mesos/pull/571#discussion_r1578088671
##########
src/slave/containerizer/mesos/isolators/cgroups2/cgroups2.hpp:
##########
@@ -149,7 +149,7 @@ class Cgroups2IsolatorProcess : public MesosIsolatorProcess
process::Future<Nothing> __cleanup(
const ContainerID& containerId,
- const std::vector<process::Future<Nothing>>& futures);
+ const process::Future<Nothing>& future);
Review Comment:
There was already a `__cleanup` function (that wasn't being used) inside of the `Cgroups2IsolatorProcess`. Therefore, rather than having to _add_ a `__cleanup` function, we just needed to update the parameters it took.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cgroups2] Update destroy to be async more robust. [mesos]
Posted by "DevinLeamy (via GitHub)" <gi...@apache.org>.
DevinLeamy commented on code in PR #571:
URL: https://github.com/apache/mesos/pull/571#discussion_r1578338470
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
+ [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
+ Try<set<pid_t>> pids = cgroups2::processes(cgroup, true);
+ if (pids.isError()) {
+ return Failure("Failed to fetch pids in cgroup: " + pids.error());
+ }
Review Comment:
Opted to convert the `.then` into a `.onAny` because I think I failure here warrants on error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cgroups2] Update destroy to be async more robust. [mesos]
Posted by "bmahler (via GitHub)" <gi...@apache.org>.
bmahler commented on code in PR #571:
URL: https://github.com/apache/mesos/pull/571#discussion_r1578387065
##########
src/slave/containerizer/mesos/isolators/cgroups2/cgroups2.cpp:
##########
@@ -751,13 +751,27 @@ Future<Nothing> Cgroups2IsolatorProcess::_cleanup(
+ strings::join(", ", errors));
}
- if (cgroups2::exists(infos[containerId]->cgroup)) {
- Try<Nothing> destroy = cgroups2::destroy(infos[containerId]->cgroup);
- if (destroy.isError()) {
- return Failure(
- "Failed to destroy cgroup '" + infos[containerId]->cgroup + "': "
- + destroy.error());
- }
+ if (!cgroups2::exists(infos[containerId]->cgroup)) {
+ return Nothing();
Review Comment:
we're not erasing from infos in this case anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cgroups2] Update destroy to be async more robust. [mesos]
Posted by "bmahler (via GitHub)" <gi...@apache.org>.
bmahler merged PR #571:
URL: https://github.com/apache/mesos/pull/571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cgroups2] Update destroy to be async more robust. [mesos]
Posted by "bmahler (via GitHub)" <gi...@apache.org>.
bmahler commented on code in PR #571:
URL: https://github.com/apache/mesos/pull/571#discussion_r1578197877
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
Review Comment:
can you reduce this to 1 millisecond?
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
+ [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
+ Try<set<pid_t>> pids = cgroups2::processes(cgroup, true);
+ if (pids.isError()) {
+ return Failure("Failed to fetch pids in cgroup: " + pids.error());
+ }
- vector<string> sorted(
- std::make_move_iterator(cgroups->begin()),
- std::make_move_iterator(cgroups->end()));
- sorted.push_back(cgroup);
- std::sort(sorted.rbegin(), sorted.rend());
+ if (pids->size() == 0) {
+ return Break();
+ }
- foreach (const string& cgroup, sorted) {
- const string path = cgroups2::path(cgroup);
- Try<Nothing> rmdir = os::rmdir(path, false);
- if (rmdir.isError()) {
- return Error(
- "Failed to remove directory '" + path + "': " + rmdir.error());
+ --retries;
+ if (retries == 0) {
+ return Failure("Failed to kill all processes in cgroup");
+ }
Review Comment:
ditto here, maybe this should just Break?
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
+ [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
+ Try<set<pid_t>> pids = cgroups2::processes(cgroup, true);
+ if (pids.isError()) {
+ return Failure("Failed to fetch pids in cgroup: " + pids.error());
+ }
- vector<string> sorted(
- std::make_move_iterator(cgroups->begin()),
- std::make_move_iterator(cgroups->end()));
- sorted.push_back(cgroup);
- std::sort(sorted.rbegin(), sorted.rend());
+ if (pids->size() == 0) {
+ return Break();
+ }
- foreach (const string& cgroup, sorted) {
- const string path = cgroups2::path(cgroup);
- Try<Nothing> rmdir = os::rmdir(path, false);
- if (rmdir.isError()) {
- return Error(
- "Failed to remove directory '" + path + "': " + rmdir.error());
+ --retries;
+ if (retries == 0) {
+ return Failure("Failed to kill all processes in cgroup");
+ }
+
+ return Continue();
+ });
+
+ return processes.then([=]() -> Future<Nothing> {
+ Try<set<string>> cgroups = cgroups2::get(cgroup);
+ if (cgroups.isError()) {
+ return Failure("Failed to get nested cgroups: " + cgroups.error());
}
- }
- return Nothing();
+ vector<string> sorted(
+ std::make_move_iterator(cgroups->begin()),
+ std::make_move_iterator(cgroups->end()));
+ sorted.push_back(cgroup);
+ std::sort(sorted.rbegin(), sorted.rend());
+
+ foreach (const string& cgroup, sorted) {
+ const string path = cgroups2::path(cgroup);
+ Try<Nothing> rmdir = os::rmdir(path, false);
Review Comment:
hm.. ideally we ignore ENOENT here, but looking at os::rmdir, it doesn't care
might be worth just using ::rmdir directly here and looking at ENOENT, since os::rmdir is mainly providing recursion support and cross OS support, which we don't care about here
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
+ [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
+ Try<set<pid_t>> pids = cgroups2::processes(cgroup, true);
+ if (pids.isError()) {
+ return Failure("Failed to fetch pids in cgroup: " + pids.error());
+ }
Review Comment:
maybe this should just Break? or if this returns Failure, then we likely want an onAny rather than a .then below, which will complicate things
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
+ [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
+ Try<set<pid_t>> pids = cgroups2::processes(cgroup, true);
+ if (pids.isError()) {
+ return Failure("Failed to fetch pids in cgroup: " + pids.error());
+ }
- vector<string> sorted(
- std::make_move_iterator(cgroups->begin()),
- std::make_move_iterator(cgroups->end()));
- sorted.push_back(cgroup);
- std::sort(sorted.rbegin(), sorted.rend());
+ if (pids->size() == 0) {
+ return Break();
+ }
Review Comment:
`pids->empty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cgroups2] Update destroy to be async more robust. [mesos]
Posted by "DevinLeamy (via GitHub)" <gi...@apache.org>.
DevinLeamy commented on code in PR #571:
URL: https://github.com/apache/mesos/pull/571#discussion_r1578339692
##########
src/linux/cgroups2.cpp:
##########
@@ -381,41 +381,66 @@ Try<Nothing> kill(const std::string& cgroup)
}
-Try<Nothing> destroy(const string& cgroup)
+Future<Nothing> destroy(const string& cgroup)
{
if (!cgroups2::exists(cgroup)) {
- return Error("Cgroup '" + cgroup + "' does not exist");
+ return Failure("Cgroup '" + cgroup + "' does not exist");
}
// To destroy a subtree of cgroups we first kill all of the processes inside
// of the cgroup and then remove all of the cgroup directories, removing
// the most deeply nested directories first.
+
Try<Nothing> kill = cgroups2::kill(cgroup);
if (kill.isError()) {
- return Error("Failed to kill processes in cgroup: " + kill.error());
+ return Failure("Failed to kill processes in cgroup: " + kill.error());
}
- Try<set<string>> cgroups = cgroups2::get(cgroup);
- if (cgroups.isError()) {
- return Error("Failed to get nested cgroups: " + cgroups.error());
- }
+ // Wait until all of the processes have been killed.
+ int retries = 50;
+ Future<Nothing> processes = loop(
+ []() { return process::after(Milliseconds(10)); },
+ [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
+ Try<set<pid_t>> pids = cgroups2::processes(cgroup, true);
+ if (pids.isError()) {
+ return Failure("Failed to fetch pids in cgroup: " + pids.error());
+ }
- vector<string> sorted(
- std::make_move_iterator(cgroups->begin()),
- std::make_move_iterator(cgroups->end()));
- sorted.push_back(cgroup);
- std::sort(sorted.rbegin(), sorted.rend());
+ if (pids->size() == 0) {
+ return Break();
+ }
- foreach (const string& cgroup, sorted) {
- const string path = cgroups2::path(cgroup);
- Try<Nothing> rmdir = os::rmdir(path, false);
- if (rmdir.isError()) {
- return Error(
- "Failed to remove directory '" + path + "': " + rmdir.error());
+ --retries;
+ if (retries == 0) {
+ return Failure("Failed to kill all processes in cgroup");
+ }
Review Comment:
I believe this should trigger a failure because otherwise we'll run into an error on `rmdir` which I think is less illustrative of the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@mesos.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org