You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Patrick Kling (JIRA)" <ji...@apache.org> on 2010/10/14 20:38:36 UTC
[jira] Created: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Allow configuration changes without restarting configured nodes
---------------------------------------------------------------
Key: HADOOP-7001
URL: https://issues.apache.org/jira/browse/HADOOP-7001
Project: Hadoop Common
Issue Type: Task
Reporter: Patrick Kling
Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
interface ChangeableConfigured extends Configured {
void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
}
The contract of changeConfiguration is as follows:
The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924842#action_12924842 ]
Alejandro Abdelnur commented on HADOOP-7001:
--------------------------------------------
I'd also prefer a refresh request that forces a reload of the config from disk.
To detect diffs a memory copy of the current config could be used.
Properties in the configuration XML should be flagged as *reconfigurable* (similar to *final*), if a property is not flagged as *reconfigurable* hot changes for that property are ignored and are not communicated via the Property listener (have you considered using the PropertyChangeListener API from the JDK?).
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921086#action_12921086 ]
Aaron T. Myers commented on HADOOP-7001:
----------------------------------------
+1 - this would be extremely useful.
May I suggest, though, that we change the name of the interface to "Reconfigurable" ?
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: HADOOP-7001.2.patch
As suggested by Doug, I have renamed the methods in Reconfigurable. I have also renamed ConfigurationChangeException to ReconfigurationException and ConfigurationChangeServlet to ReconfigurationServlet to keep everything consistent.
Please see HDFS-1477 for a patch that makes NameNode Reconfigurable and allows us to change 2 properties.
The servlet now returns error code 500 if a change fails.
A command line tool for changing configuration properties sounds like a good idea but I am not quite sure how to provide a generic version of such a tool. If the command line tool runs locally, very little functionality beyond calling the interface methods would be needed. Maybe it would be best to leave command line utilities to a later JIRA once it becomes more clear if there is a need for additional support.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: reconfigurable.patch
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: HADOOP-7001.3.patch
Here is an updated patch. Navigating to the ReconfigurationServlet for a Reconfigurable node now reloads the configuration files from disk and shows a list of all properties that have changed (along with their old and new values). If there are any changes to properties that cannot be reconfigured at run time, a warning is shown. After reviewing the changes, the admin can approve them, at which point they are applied through the methods in the Reconfigurable interface.
The code for computing the diff between the old and the new configuration is in ReconfigurationUtil, which should make it easy to add this functionality to existing command line tools.
For this revision, I didn't end up going with the PropertyChangeListener API, since there doesn't seem to be an ideal source object with which to register listeners while still keeping this change generic enough to be useful for all configured nodes in the system.
The review board revision corresponding to this patch can be found at: https://reviews.apache.org/r/21/
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.2.patch, HADOOP-7001.3.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921552#action_12921552 ]
Patrick Kling commented on HADOOP-7001:
---------------------------------------
I absolutely agree that there will likely be many configuration options that cannot be changed at run time and implementing Reconfigurable does not require that we can make arbitrary changes. For the configuration changes that are possible at run time (such as the existing examples mentioned by Philip), it would be nice to have a consistent mechanism. The interface could also make it easier to propagate configuration changes (by calling changeConf on objects to which the current configuration was propagated).
This should be a bit easier to read than the JavaDoc I posted before:
{code}
package org.apache.hadoop.conf;
/**
* Something whose {@link Configuration} can be changed at run time.
*/
public interface Reconfigurable extends Configurable {
/**
* Change the configuration of this object.
*
* Change the configuration of this object to the configuration
* passed as conf.
* If it is not possible to change a configuration option,
* a {@link ConfigurationChangeException} is thrown
* and no changes are made to the current configuration
*/
void changeConf(Configuration conf) throws ConfigurationChangeException;
}
{code}
{code}
package org.apache.hadoop.conf;
/**
* Exception indicating that configuration property cannot be changed
* at run time.
*/
public class ConfigurationChangeException extends Exception {
private String property;
private String newVal;
private String oldVal;
/**
* Construct the exception message.
*/
private static String constructMessage(String property,
String newVal, String oldVal) {
String message = "Could not change property " + property;
if (oldVal != null) {
message += " from " + oldVal;
}
if (newVal != null) {
message += " to " + newVal;
}
return message;
}
/**
* Create a new instance of {@link ConfigurationChangeException}.
*/
public ConfigurationChangeException(String property,
String newVal, String oldVal) {
super(constructMessage(property, newVal, oldVal));
this.property = property;
this.newVal = newVal;
this.oldVal = oldVal;
}
/**
* Create a new instance of {@link ConfigurationChangeException}.
*/
public ConfigurationChangeException(String property,
String newVal, String oldVal,
Throwable cause) {
super(constructMessage(property, newVal, oldVal), cause);
this.property = property;
this.newVal = newVal;
this.oldVal = oldVal;
}
/**
* Get property that cannot be changed.
*/
public String getProperty() {
return property;
}
/**
* Get value to which property was supposed to be changed.
*/
public String getNewValue() {
return newVal;
}
/**
* Get old value of property that cannot be changed.
*/
public String getOldValue() {
return oldVal;
}
{code}
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921133#action_12921133 ]
Patrick Kling commented on HADOOP-7001:
---------------------------------------
org.apache.hadoop.conf
Interface Reconfigurable
All Superinterfaces:
Configurable
___________________________________________________________________________________________________________________________________________
public interface Reconfigurable
extends Configurable
Something whose Configuration can be changed at run time.
___________________________________________________________________________________________________________________________________________
Method Summary
void changeConf(Configuration conf)
changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is
thrown and no changes are made to the current configuration
Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf
Method Detail
changeConf
void changeConf(Configuration conf)
throws ConfigurationChangeException
changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is thrown
and no changes are made to the current configuration
Throws:
ConfigurationChangeException
Class ConfigurationChangeException
java.lang.Object
extended by java.lang.Throwable
extended by java.lang.Exception
extended by org.apache.hadoop.conf.ConfigurationChangeException
All Implemented Interfaces:
Serializable
___________________________________________________________________________________________________________________________________________
public class ConfigurationChangeException
extends Exception
exception indicating that configuration property cannot be changed at run time
See Also:
Serialized Form
___________________________________________________________________________________________________________________________________________
Constructor Summary
ConfigurationChangeException(String property)
Creates a new instance of ConfigurationChangeException
ConfigurationChangeException(String property, String newVal)
Creates a new instance of ConfigurationChangeException
ConfigurationChangeException(String property, String newVal, String oldVal)
Creates a new instance of ConfigurationChangeException
Method Summary
String getMessage()
gets message describing exception
String getNewValue()
gets value to which property was supposed to be changed
String getOldValue()
gets old value of property that cannot be changed
String getProperty()
gets property that cannot be changed
Methods inherited from class java.lang.Throwable
fillInStackTrace, getCause, getLocalizedMessage, getStackTrace, initCause, printStackTrace, printStackTrace, printStackTrace, setStackTrace, toString
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Constructor Detail
ConfigurationChangeException
public ConfigurationChangeException(String property,
String newVal,
String oldVal)
Creates a new instance of ConfigurationChangeException
___________________________________________________________________________________________________________________________________________
ConfigurationChangeException
public ConfigurationChangeException(String property,
String newVal)
Creates a new instance of ConfigurationChangeException
___________________________________________________________________________________________________________________________________________
ConfigurationChangeException
public ConfigurationChangeException(String property)
Creates a new instance of ConfigurationChangeException
getProperty
public String getProperty()
gets property that cannot be changed
___________________________________________________________________________________________________________________________________________
getNewValue
public String getNewValue()
gets value to which property was supposed to be changed
___________________________________________________________________________________________________________________________________________
getOldValue
public String getOldValue()
gets old value of property that cannot be changed
___________________________________________________________________________________________________________________________________________
getMessage
public String getMessage()
gets message describing exception
Overrides:
getMessage in class Throwable
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921553#action_12921553 ]
Philip Zeyliger commented on HADOOP-7001:
-----------------------------------------
I think that the number of reconfigurable parameters will be quite small, and that they will require extra code. (e.g., changing the size of a thread pool requires a line or two of code). So maybe it's just better to expose those things in some more direct way? At the very least, we should make it quite clear where it's going to be documented what things are re-configurable.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-7001:
----------------------------------------
Assignee: Patrick Kling
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: (was: reconfigurable.patch)
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921493#action_12921493 ]
Philip Zeyliger commented on HADOOP-7001:
-----------------------------------------
I might be grumpy, but I think the right way to deal with configuration changes in a distributed, fault-tolerant system is to just restart the daemon entirely. We already have to deal with the daemon suddenly crashing and that not affecting the system too much, so I'm wary of extra complexity of yet another process. In practice, many configuration variables are loaded at start time and then stored as statics: stuff like threadpool sizes, for example. Not to mention that Configuration objects get copied along, so it's hard to make sure that a configuration change propagates to all possible children.
I'll point out that the namenode and the jobtracker's fair scheduler already have mechanisms for dynamic configuration changes. In namenode, that's -refreshNodes. In the jt, I think the fair scheduler re-reads its XML configuration file on occasion. Both of these make a lot of sense: these are specific endpoints for managing specific data, and the semantics of those changes are easy to understand.
-- Philip
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: reconfigurable.patch
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: HADOOP-7001.4.patch
Incorporated Todd's comments on the review board.
- Added unit tests for ReconfigurationUtil.getChangedProperties, Reconfigurable interface methods and cross-thread visibility of configuration changes.
- Added proper escaping to servlet.
- Removed synchronization from Configuration methods that were already protected by synchronization of getProps(). In this revision, I have only added synchronization to the setter methods.
- Changed getReconfigurableProperties to return Collection rather than List.
- Fixed typo.
- Removed commented-out code.
- Removed unnecessary imports.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.2.patch, HADOOP-7001.3.patch, HADOOP-7001.4.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HADOOP-7001:
----------------------------------
Attachment: HADOOP-7001.patch
Here is a new version of my patch. I have changed the interface Reconfigurable somewhat: Since only a subset of configuration properties can be changed at run time, it makes more sense to have an interface that allows us to change a single property at a time, rather than passing a Configuration object and then detecting all changes that need to be made. I have also added methods for detecting which configuration properties can be changed at runtime.
{code}
/**
* Something whose {@link Configuration} can be changed at run time.
*/
public interface Reconfigurable extends Configurable {
/**
* Change a the configuration property on this object to the value specified.
*
* Change a the configuration property on this object to the value specified
* and return the previous value that the configuration property was set to
* (or null if it was not previously set).
*
* If the property cannot be changed, throw a
* {@link ConfigurationChangeException}.
*/
public String changeConfProperty(String property, String newVal)
throws ConfigurationChangeException;
/**
* Return whether a given property is changeable at run time.
*
* If isConfPropertyChangeable returns true for a property,
* then changeConf should not throw an exception when changing
* this property.
*/
public boolean isConfPropertyChangeable(String property);
/**
* Return all the properties that can be changed at run time.
*/
public List<String> getChangeableConfProperties();
}
{code}
The patch also includes a simple servlet for changing the configuration of a Reconfigurable. In order to allow configuration changes at run time, I made all the getters and setters in Configuration synchronized.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921544#action_12921544 ]
dhruba borthakur commented on HADOOP-7001:
------------------------------------------
But a simple thing like increasing the number of handler threads in a server require the downtime of the server. If we can make the namenode implement Reconfigurable, then we will have the ability to change a few select of parameters dynamically, isn't it?
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924755#action_12924755 ]
Doug Cutting commented on HADOOP-7001:
--------------------------------------
It doesn't seem unreasonable to me for some limited changes might be permitted without restarting. Restarting a namenode is not a lightweight operation.
If the interface is 'Reconfigurable' then the method might be named 'reconfigureProp(String,String)', no?
I don't like adding new interfaces that aren't yet used. So it might be good to file issues for at least one the daemons that you expect to implement this and at least draft those patches before this is committed. If we can't agree on some concrete usages of this interface within Hadoop, then there's no point in adding it.
The servlet should probably return an HTTP error code when reconfiguration fails.
A command-line client might be useful. Perhaps this will be added to, e.g.. the existing HDFS admin commands. But each command-line tool that uses this will share client-side HTTP call, handling of error codes, etc. So perhaps that should be encapsulated in a shared utility.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925684#action_12925684 ]
dhruba borthakur commented on HADOOP-7001:
------------------------------------------
I too like the idea of having "bin/hadoop dfs -refreshConfig" that will reload the config from the conf file, diff the newly read ones with the ones in the NameNode.
If the only ones that are different are Recofinfigurable, then the refreshConfig command will be successful, otherwise it is an error. The servlet could take a similar approach.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921102#action_12921102 ]
Patrick Kling commented on HADOOP-7001:
---------------------------------------
I agree, Reconfigurable sounds better. Also, we would be extending the interface Configurable (not the class Configured, I got confused there). I also wanted to point out that Reconfigurable would introduce the new method changeConf in addition to the method setConf from Configurable because there is a difference in semantics:
changeConf would make sure that all changes immediately affect the behaviour of the node (or failing that an exception would be thrown), whereas setConf as it is used now does not give any such guarantee (since some configuration values might be cached in other variables and some behaviour might not be changeable without a restart).
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924738#action_12924738 ]
dhruba borthakur commented on HADOOP-7001:
------------------------------------------
I added Doug and Tom to the list of watchers. I am hoping that they provide some feedback to this JIRA.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921128#action_12921128 ]
dhruba borthakur commented on HADOOP-7001:
------------------------------------------
It would be nice if you can paste the javadocs for this new interface and the method(s) in it. This will help understand the new API.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924832#action_12924832 ]
Todd Lipcon commented on HADOOP-7001:
-------------------------------------
I'm not sold on the interface being one of "set conf var 'foo' to value 'bar'". Internally, this seems fine, but exposing this as a servlet seems like it will encourage the case where runtime configuration diverges from what's in the XML configuration files on disk.
Could we instead have the servlet/API be more similar to the existing refresh* functions, where it reloads the configuration off disk, and applies a diff? I'm imagining the servlet would show you a "preview" of the diff to be applied, along with a list of warnings for configuration variables that have been changed on disk but are *not* reconfigurable. Then the administrator can hit an "apply" button to reconfigure the settings that are reconfigurable.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921106#action_12921106 ]
Konstantin Boudnik commented on HADOOP-7001:
--------------------------------------------
Also, this Herriot system testing framework will be highly benefited from such a change. +1 on the idea.
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7001) Allow configuration changes without
restarting configured nodes
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921584#action_12921584 ]
Patrick Kling commented on HADOOP-7001:
---------------------------------------
I have uploaded a patch with the changes proposed in this JIRA. The included base class can be used to easily implement the interface by overriding the method changeProperty, which modifies a single configuration property:
{code}
/**
* Change a property or throw an exception.
*
* Subclasses should overrride this.
* If newVal is null, change the property to its default value.
*/
protected boolean changeProperty(String property, String newVal)
throws ConfigurationChangeException {
throw new ConfigurationChangeException(property, newVal,
getConf().get(property));
}
{code}
This should enable us to include whatever code is necessary to make a configuration change. The overridden version of this method should also serve as a good documentation of what changes are supported by a particular class.
Below is a version of the Reconfigurable interface with more detailed comments. Unfortunately, I cannot edit my previous comments, so I apologize if this is somewhat redundant:
{code}
package org.apache.hadoop.conf;
/**
* Something whose {@link Configuration} can be changed at run time.
*/
public interface Reconfigurable extends Configurable {
/**
* Change the configuration of this object.
*
* Change the configuration of this object to the configuration
* passed as conf.
* If it is not possible to change a configuration option,
* a {@link ConfigurationChangeException} is thrown
* and no changes are made to the current configuration.
*
* Detailed semantics:
* * If a property is set in the current configuration and in newConf, then
* the configuration must be updated to reflect the value in newConf.
* * If a property is not set in the current configuration but is set in
* newConf, then the configuration must be set to the value in newConf.
* * If a property is set in the current configuration but is not set in
* newConf, then the configuration must revert to the default value for
* this property.
* * If the change required by these rules is not possible, a
* ConfigurationChangeException must be thrown.
*/
void changeConf(Configuration newConf) throws ConfigurationChangeException;
}
{code}
> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
> Key: HADOOP-7001
> URL: https://issues.apache.org/jira/browse/HADOOP-7001
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
> void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.