You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by bm...@apache.org on 2019/10/04 19:23:48 UTC
[mesos] branch 1.7.x updated (94321d8 -> 71685b5)
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a change to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 94321d8 Added MESOS-9966 to the 1.7.3 CHANGELOG.
new 7e036af Fixed master::Slave::hasExecutor performance issue.
new ff35e2f Eliminate accidental loop amplification over unreachable tasks.
new a1e54d0 Added MESOS-9948 to the 1.7.3 CHANGELOG.
new 71685b5 Added MESOS-9889 to the 1.7.3 CHANGELOG.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG | 2 ++
src/master/master.cpp | 18 +++++++++---------
src/master/master.hpp | 3 ++-
3 files changed, 13 insertions(+), 10 deletions(-)
[mesos] 02/04: Eliminate accidental loop amplification over
unreachable tasks.
Posted by bm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit ff35e2faccf07efe48f926736e2c8020c0e3a478
Author: Benjamin Mahler <bm...@apache.org>
AuthorDate: Thu Oct 3 16:39:51 2019 -0400
Eliminate accidental loop amplification over unreachable tasks.
Per MESOS-9889, the foreachkey operator on a multimap will actually
loop over each <key,value> entry, thus looping over the same key
multiple times. The code in the master is written such that
excessive looping occurs.
For example: <F1,T1> <F1,T2> <F1,T3>
This will consider T1,T2,T3 3 times each rather than once each!
We could fix this using .keys(), but this patch just replaces the
multimap with a map of vectors.
Review: https://reviews.apache.org/r/71579
---
src/master/master.cpp | 16 ++++++++--------
src/master/master.hpp | 3 ++-
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 980ce17..1f7dd3a 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -1949,7 +1949,7 @@ void Master::_doRegistryGc(
Framework* framework = getFramework(frameworkId);
if (framework != nullptr) {
foreach (const TaskID& taskId,
- slaves.unreachableTasks.at(slaveId).get(frameworkId)) {
+ slaves.unreachableTasks.at(slaveId).at(frameworkId)) {
framework->unreachableTasks.erase(taskId);
}
}
@@ -7525,12 +7525,12 @@ void Master::__reregisterSlave(
// All tasks from this agent are now reachable so clean them up from
// the master's unreachable task records.
if (slaves.unreachableTasks.contains(slaveInfo.id())) {
- foreachkey (FrameworkID frameworkId,
- slaves.unreachableTasks.at(slaveInfo.id())) {
+ foreachkey (const FrameworkID& frameworkId,
+ slaves.unreachableTasks.at(slaveInfo.id())) {
Framework* framework = getFramework(frameworkId);
if (framework != nullptr) {
- foreach (TaskID taskId,
- slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) {
+ foreach (const TaskID& taskId,
+ slaves.unreachableTasks.at(slaveInfo.id()).at(frameworkId)) {
framework->unreachableTasks.erase(taskId);
}
}
@@ -9018,7 +9018,7 @@ void Master::markGone(const SlaveID& slaveId, const TimeInfo& goneTime)
}
foreach (const TaskID& taskId,
- slaves.unreachableTasks.at(slaveId).get(frameworkId)) {
+ slaves.unreachableTasks.at(slaveId).at(frameworkId)) {
if (framework->unreachableTasks.contains(taskId)) {
const Owned<Task>& task = framework->unreachableTasks.at(taskId);
@@ -11277,8 +11277,8 @@ void Master::removeTask(Task* task, bool unreachable)
}
if (unreachable) {
- slaves.unreachableTasks[slave->id].put(
- task->framework_id(), task->task_id());
+ slaves.unreachableTasks[slave->id][task->framework_id()]
+ .push_back(task->task_id());
}
// Remove from framework.
diff --git a/src/master/master.hpp b/src/master/master.hpp
index b5d82ee..1368ff9 100644
--- a/src/master/master.hpp
+++ b/src/master/master.hpp
@@ -2054,7 +2054,8 @@ private:
// agent reregisters. This map is bounded by the same GC behavior as
// `unreachable`. When the agent is GC'd from unreachable it's also
// erased from `unreachableTasks`.
- hashmap<SlaveID, multihashmap<FrameworkID, TaskID>> unreachableTasks;
+ hashmap<SlaveID, hashmap<FrameworkID, std::vector<TaskID>>>
+ unreachableTasks;
// Slaves that have been marked gone. We recover this from the
// registry, so it includes slaves marked as gone by other instances
[mesos] 01/04: Fixed master::Slave::hasExecutor performance issue.
Posted by bm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 7e036afedcba87ee5365c399da884643a5e5497f
Author: Benjamin Mahler <bm...@apache.org>
AuthorDate: Fri Oct 4 13:52:20 2019 -0400
Fixed master::Slave::hasExecutor performance issue.
This is a backport of the broader fix in MESOS-9948.
---
src/master/master.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index 1a95b69..980ce17 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -12657,7 +12657,7 @@ bool Slave::hasExecutor(const FrameworkID& frameworkId,
const ExecutorID& executorId) const
{
return executors.contains(frameworkId) &&
- executors.get(frameworkId)->contains(executorId);
+ executors.at(frameworkId).contains(executorId);
}
[mesos] 04/04: Added MESOS-9889 to the 1.7.3 CHANGELOG.
Posted by bm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 71685b5c807e4d4d1a5ebe33a3c3c0a2f824b9b4
Author: Benjamin Mahler <bm...@apache.org>
AuthorDate: Fri Oct 4 14:03:38 2019 -0400
Added MESOS-9889 to the 1.7.3 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index d7363d5..b83b22a 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -30,6 +30,7 @@ Release Notes - Mesos - Version 1.7.3 (WIP)
* [MESOS-9868] - NetworkInfo from the agent /state endpoint is not correct.
* [MESOS-9870] - Simultaneous adding/removal of a role from framework's roles and its suppressed roles crashes the master.
* [MESOS-9887] - Race condition between two terminal task status updates for Docker/Command executor.
+ * [MESOS-9889] - Master CPU high due to unexpected foreachkey behaviour in Master::__reregisterSlave.
* [MESOS-9893] - `volume/secret` isolator should cleanup the stored secret from runtime directory when the container is destroyed.
* [MESOS-9925] - Default executor takes a couple of seconds to start and subscribe Mesos agent.
* [MESOS-9966] - Agent crashes when trying to destroy orphaned nested container if root container is orphaned as well.
[mesos] 03/04: Added MESOS-9948 to the 1.7.3 CHANGELOG.
Posted by bm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bmahler pushed a commit to branch 1.7.x
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit a1e54d000ee3ea587ef1e9ee52be01605796396b
Author: Benjamin Mahler <bm...@apache.org>
AuthorDate: Fri Oct 4 13:58:46 2019 -0400
Added MESOS-9948 to the 1.7.3 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index eda1040..d7363d5 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -41,6 +41,7 @@ Release Notes - Mesos - Version 1.7.3 (WIP)
* [MESOS-9675] - Docker Manifest V2 Schema2 Support.
* [MESOS-9704] - Support docker manifest v2s2 config GC.
* [MESOS-9759] - Log required quota headroom and available quota headroom in the allocator.
+ * [MESOS-9948] - master::Slave::hasExecutor occupies 37% of a 150 second perf sample.
Release Notes - Mesos - Version 1.7.2