You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by "Matei Zaharia (Created) (JIRA)" <ji...@apache.org> on 2011/12/15 21:57:30 UTC
[jira] [Created] (MESOS-106) Failover timeout should default to 0
Failover timeout should default to 0
------------------------------------
Key: MESOS-106
URL: https://issues.apache.org/jira/browse/MESOS-106
Project: Mesos
Issue Type: Improvement
Reporter: Matei Zaharia
Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-106) Failover timeout should default to 0
Posted by "Benjamin Hindman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172516#comment-13172516 ]
Benjamin Hindman commented on MESOS-106:
----------------------------------------
LGTM.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-106) Failover timeout should default to 0
Posted by "Matei Zaharia (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MESOS-106:
--------------------------------
Attachment: MESOS-106.patch
Here's a patch for this (very simple). Let me know if it looks OK.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-106) Failover timeout should default to 0
Posted by "Benjamin Hindman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170508#comment-13170508 ]
Benjamin Hindman commented on MESOS-106:
----------------------------------------
This sounds fine with me ... but it will require the failover timeout to be made configurable first.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-106) Failover timeout should default to 0
Posted by "Matei Zaharia (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MESOS-106:
--------------------------------
Attachment: MESOS-106-v2.patch
Sure, here's an updated patch.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106-v2.patch, MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MESOS-106) Failover timeout should default to 1
Posted by "Matei Zaharia (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia resolved MESOS-106.
---------------------------------
Resolution: Fixed
Just committed this after Ben OK'ed it.
> Failover timeout should default to 1
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106-v2.patch, MESOS-106-v3.patch, MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 1, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-106) Failover timeout should default to 0
Posted by "Matei Zaharia (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170832#comment-13170832 ]
Matei Zaharia commented on MESOS-106:
-------------------------------------
It is configurable actually -- I added that a while back. We just need to change the default value.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-106) Failover timeout should default to 1
Posted by "Matei Zaharia (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MESOS-106:
--------------------------------
Description: Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 1, because almost no users have failover-enabled frameworks. (was: Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.)
Summary: Failover timeout should default to 1 (was: Failover timeout should default to 0)
> Failover timeout should default to 1
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106-v2.patch, MESOS-106-v3.patch, MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 1, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-106) Failover timeout should default to 0
Posted by "Matei Zaharia (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MESOS-106:
--------------------------------
Attachment: MESOS-106-v3.patch
Actually some users have reported that a timeout of 0 doesn't always work, so I'm going to set it to 1. I think this might be a timing issue if you schedule an event for the current time in libprocess, but I'm not sure.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106-v2.patch, MESOS-106-v3.patch, MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-106) Failover timeout should default to 0
Posted by "Benjamin Hindman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172520#comment-13172520 ]
Benjamin Hindman commented on MESOS-106:
----------------------------------------
Actually, it would be sweet if we made that a constant in master/constants.hpp so that all constants are defined there instead of at the actual configuration 'get' sites.
> Failover timeout should default to 0
> ------------------------------------
>
> Key: MESOS-106
> URL: https://issues.apache.org/jira/browse/MESOS-106
> Project: Mesos
> Issue Type: Improvement
> Reporter: Matei Zaharia
> Attachments: MESOS-106.patch
>
>
> Since the failover timeout was added, you get a lot of weird behavior in clusters running frameworks that don't support failover due to its long default value of 1 day. If a framework fails or just exits without calling driver.stop(), all its executors stay around and consume resources on the machines, causing subsequent runs to mysteriously fail to acquire resources. See http://groups.google.com/group/spark-users/msg/553af12424e4ed3d for an example. I know that the failover timeout is supposed to eventually become a per-framework parameter anyway, but in the meantime, the easiest way to prevent this is to set it to 0, because almost no users have failover-enabled frameworks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira