You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Carlos Valiente (JIRA)" <ji...@apache.org> on 2009/02/13 19:43:00 UTC
[jira] Created: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Export namenode/datanode functionality through a pluggable RPC layer
--------------------------------------------------------------------
Key: HADOOP-5257
URL: https://issues.apache.org/jira/browse/HADOOP-5257
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Carlos Valiente
Priority: Minor
Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
Plug-in implementations would extend this base class:
{code}abstract class Plugin {
public abstract datanodeStarted(DataNode datanode);
public abstract datanodeStopping();
public abstract namenodeStarted(NameNode namenode);
public abstract namenodeStopping();
}{code}
Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
{code}public class NameNode {
// [..]
private void initialize(Configuration conf)
// [...]
for (Plugin p: PluginManager.loadPlugins(conf))
p.namenodeStarted(this);
}
// [..]
public void stop() {
if (stopRequested)
return;
stopRequested = true;
for (Plugin p: plugins)
p.namenodeStopping();
// [..]
}
// [..]
}{code}
Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v6.patch
bq. Plugin might better be called ServicePlugin [..]
Done in HADOOP-5257-v6.patch, Doug --- and moved to {{org.apache.hadoop.util}} under {{src/core}}. I also added some null checks on {{DataNode.shutdown()}} and {{NameNode.stop()}} to fix several test failures.
'ant test-core' and 'ant test-patch' succeed.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674842#action_12674842 ]
Doug Cutting commented on HADOOP-5257:
--------------------------------------
It seems likely we'll want plugins elsewhere, so the manager might be more abstract, able to return instances of other interfaces determined by different configuration properties. This might go into core. Then we might have separate datanode and namenode plugin interfaces and configuration properties.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688826#action_12688826 ]
Steve Loughran commented on HADOOP-5257:
----------------------------------------
@Dhruba
yes, we can defer the others until later.
Other points
# which classloader is being used to load classes?
# If parsing a string value to a list is useful, this should really go into Configuration, not the plugin classes, as that is one places to implement string trim policy, write the unit tests, etc.
# I like the tests, add one to try loading a class that isn't there
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694201#action_12694201 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
Code looks good. +1.
The test failures are not related to this patch.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5257:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Assignee: Carlos Valiente
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Carlos Valiente
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682388#action_12682388 ]
Konstantin Shvachko commented on HADOOP-5257:
---------------------------------------------
Minor comments:
# DFS should be HDFS
# It is not clear from the name TestGetInstances what it tests and there is no JavaDoc explaining it.
# It would be better if classes ChildInterface, SampleInterface, AnotherClass, SampleClass were static subclasses of TestGetInstances.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v2.patch
Thanks for the feedback, guys. HADOOP-5257-v2.patch implements {{Configuration.getInstances()}}, as suggested by Doug. Datanode and namenode interfaces are now specified by two distinct classes, and the start and stop order is mentioned in both clasess' javadocs, as suggested by Dhruba.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688226#action_12688226 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
+1 for this code.
Steve: do you see this patch going into trunk (0.21) without stepping on the toes for HADOOP-3628?
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v5.patch
Oops, wrong file - HADOOP-5257-v5.patch is the right one
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Release Note: Added support for HDFS pluggable components.
Status: Patch Available (was: Open)
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682461#action_12682461 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
> While modifying the javadoc for org.apache.hadoop.hdfs.server.protocol.Plugin, I've noticed there's nothing HDFS-specific to that interface - should it perhaps be moved to some place under src/core instead?
Yes, that sounds right. It seems like a good idea to move it to core.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674877#action_12674877 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
@Doug: yes, we can make the plugin more general and make it part of org.apache.hadoop.util package. Can you pl enumerate the APIs (other than start/stop) that this interface might provide?
@Carlos: the namenode/datanode plugin interface javadoc may want to document that the plugins are started after the namenode is started and is stopped before the namenode is stopped.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683800#action_12683800 ]
Carlos Valiente commented on HADOOP-5257:
-----------------------------------------
Thanks for your comments, Steve.
bq.1. Looking at the code, the plugin interface is very like the service lifecycle in HADOOP-3628; start, stop, and such like. There's no reason why we couldn't use the same lifecycle interface for plugins as the services themselves
Sounds very reasonable. Just one thing: For my current prototype of the Thrift Namenode plugin, I use the reference to the {{o.a.h.hdfs.server.namenode.NameNode}} instance passed as the parameter to the {{ServicePlugin.start(server)}}. Something like this:
{code}
class NamenodePlugin implements ServicePlugin {
NameNode namenode;
void start(Object server) {
this.namenode = (NameNode)server;
}
// [..]
/* A sample method of the Thrift interface */
public void enterSafeMode() {
namenode.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
}
{code}
How could that be achieved in the context of HADOOP-3628?
All in all I'm happy to base this patch on HADOOP-3628. Are you aiming for 0.21 with your stuff, Steve? If HADOOP-3628 were to be included in 0.21, that would make things very easy for my Thrift stuff.
bq.2. when you aggregate, you soon reach the policy questions: how to handle failure on startup. This patch just warns and continues. I may not want my name nodes to start up if they cant start a critical plugin
What about having configurable options, like {{dfs.namenode.plugin.failonerror}}, and {{dfs.dataenode.plugin.failonerror}}, so that each service may decide what to do?
bq.3. Also: configuration. How do these plugins get configured. Are they expected to sneak a look at their parent's configuration [..]
In its current form, yes, they may have a look at their parent's config (since they get a reference to the parent in the call to {{ServicePlugin.start(server)}}. Each plugin may also look for configuration objects somewhere else via {{Configuration.addResource()}}.
In the context of HADOOP-3628 I suppose their parents would use something like {{plugin.setConf(this.conf)}}.
bq.4. Finally if the plugin interface extends Closeable(), it would make it possible for us to have utility code to handle sets of closeable items for cleanup, usable in lots of other places too.
Good idea
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688851#action_12688851 ]
Carlos Valiente commented on HADOOP-5257:
-----------------------------------------
bq.My other issue is that there are some very hard coded assumption that this code is limited to the filesystem. As doug says, a ServicePlugin would be more generic. You'd have to have separate classes for the NamenodeServicePlugin and the DatanodeServicePlugin, but you wouldn't need more interfaces when you came to add support for TaskTracker, JobTracker, Pig, etc.
The attachment [^HADOOP-5257-v8.patch] already addresses that, I think
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688358#action_12688358 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
> 1. Making all these things Closeable() would be handy,
This makes sense. Carlos: would you be able to make this change to the current patch?
> 2. I'm unsure about the failure handling here when there is a chain of plugged in things.
I agree. How about if we keep it simple and say that it will be logged. There is no guarantee on the order of plugins startup sequence either.
> 3. We could use the ping operation in HADOOP-3628,
> 4. If the plugins took an implementation of Service in their constructor,...
> 5. We could extend the MockService to do some plugin
The above three things woudl be great, maybe we can defer these three things till HADOOP-3628 makes it to trunk?
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693909#action_12693909 ]
Hadoop QA commented on HADOOP-5257:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12403497/HADOOP-5257-v8.patch
against trunk revision 759932.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/79/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/79/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/79/console
This message is automatically generated.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v7.patch
Minor fixes for 'ant checkstyle'
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v8.patch
bq.This makes sense. Carlos: would you be able to make this change to the current patch?
{{ServicePlugin}} in patch HADOOP-5257-v8.patch now extends {{java.io.Closeable}}.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693911#action_12693911 ]
Carlos Valiente commented on HADOOP-5257:
-----------------------------------------
I don't think test failures are related to this patch:
{code}
java.lang.NullPointerException
at org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2117)
at org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2153)
at org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:2221)
at org.apache.hadoop.mapred.TestCapacityScheduler$FakeTaskTrackerManager.killJob(TestCapacityScheduler.java:359)
at org.apache.hadoop.mapred.TestCapacityScheduler.testClusterBlockingForLackOfMemory(TestCapacityScheduler.java:2069)
{code}
and:
{code}
java.lang.NullPointerException
at org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2117)
at org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2153)
at org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:2221)
at org.apache.hadoop.mapred.TestCapacityScheduler$FakeTaskTrackerManager.killJob(TestCapacityScheduler.java:359)
at org.apache.hadoop.mapred.CapacityTaskScheduler.killJobIfInvalidRequirements(CapacityTaskScheduler.java:1431)
at org.apache.hadoop.mapred.CapacityTaskScheduler.jobAdded(CapacityTaskScheduler.java:1463)
at org.apache.hadoop.mapred.JobQueuesManager.jobAdded(JobQueuesManager.java:183)
at org.apache.hadoop.mapred.TestCapacityScheduler$FakeTaskTrackerManager.submitJob(TestCapacityScheduler.java:387)
at org.apache.hadoop.mapred.TestCapacityScheduler.submitJob(TestCapacityScheduler.java:625)
at org.apache.hadoop.mapred.TestCapacityScheduler.testHighMemoryJobWithInvalidRequirements(TestCapacityScheduler.java:1992)
{code}
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695249#action_12695249 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
I have committed this patch. Thanks Carlos!
Can some admin please re-assign this patch to Carlos?
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257.patch
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683670#action_12683670 ]
Steve Loughran commented on HADOOP-5257:
----------------------------------------
1. Looking at the code, the plugin interface is very like the service lifecycle in HADOOP-3628; start, stop, and such like. There's no reason why we couldn't use the same lifecycle interface for plugins as the services themselves; they just push their children through the same lifecycle. The ping() operation could be used to aggregate the state of the children; you could even model the http and ipc servers similarly. I've been contemplating doing this for MiniMRCluster.
2. when you aggregate, you soon reach the policy questions: how to handle failure on startup. This patch just warns and continues. I may not want my name nodes to start up if they cant start a critical plugin; having it drop to the failed state with a stack trace that management tools can see. At the very least, the warn/fail choice should be an option for the various nodes; having plugin-specific policy is feature creep.
3. Also: configuration. How do these plugins get configured. Are they expected to sneak a look at their parent's configuration, or should the parent pass down a specific configuration for them?
4. Finally if the plugin interface extends Closeable(), it would make it possible for us to have utility code to handle sets of closeable items for cleanup, usable in lots of other places too
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5257:
-------------------------------------
Status: Open (was: Patch Available)
Cancelling patch to re-trigger hudsonqa tests
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: (was: HADOOP-5257-v4.patch)
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682589#action_12682589 ]
Carlos Valiente commented on HADOOP-5257:
-----------------------------------------
bq. Moving Plugin to core means that you will not be able to call HDFS specific methods in unit tests you will probably write later for testing.
My idea was to move just the {{Plugin}} interface to core, and keep the implementation classes in HADOOP-4707 (and their unit tests) under the appropriate HDFS packages --- but perhaps it's just simpler to leave {{Plugin}} where it lives now. Any suggestions?
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v4.patch
bq. Since these are kinda-APIs to namenode/datanode, they should probably reside in src/hdfs/org/apache/hadoop/hdfs/server/protocol.
Attachement HADOOP-5257-v4.patch includes that, Dhruba.
bq. Also, there could be just one Interface (instead of two), both the namenode and datanode should be able to use the same interface. The Interface would have two methods, start and stop.
Included as well:
{code}
public interface Plugin {
public void start(Object server);
public void stop();
}
{code}
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695432#action_12695432 ]
Hudson commented on HADOOP-5257:
--------------------------------
Integrated in Hadoop-trunk #796 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/796/])
. HDFS servers may start and stop external components through
a plugin interface. (Carlos Valiente via dhruba)
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682512#action_12682512 ]
Konstantin Shvachko commented on HADOOP-5257:
---------------------------------------------
Moving Plugin to core means that you will not be able to call HDFS specific methods in unit tests you will probably write later for testing.
I don't know whether you will need that, just a consideration to take into account.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5257:
-------------------------------------
Status: Patch Available (was: Open)
Trigger hadoopQA tests
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682726#action_12682726 ]
Doug Cutting commented on HADOOP-5257:
--------------------------------------
Plugin might better be called ServicePlugin, since, while not HDFS-specific, it is specific to services that are started and stopped. If it were so renamed, I'd support moving it to src/core, where it might nicely complement HADOOP-3628.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v4.patch
Thanks for your feedback, Konstantin
bq. 1. DFS should be HDFS
Yep - done in HADOOP-5257-v4.patch
bq. 2. It is not clear from the name TestGetInstances what it tests and there is no JavaDoc explaining it.
I've added a javadoc comment explaining the purpose of the test.
bq. 3. It would be better if classes ChildInterface, SampleInterface, AnotherClass, SampleClass were static subclasses of TestGetInstances.
Done.
While modifying the javadoc for {{org.apache.hadoop.hdfs.server.protocol.Plugin}}, I've noticed there's nothing HDFS-specific to that interface - should it perhaps be moved to some place under {{src/core}} instead?
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v4.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688282#action_12688282 ]
Steve Loughran commented on HADOOP-5257:
----------------------------------------
@Druba. The stuff could co-exist, I'm just thinking about how to do it most cleanly
# Making all these things Closeable() would be handy, as there is utility stuff common to lots of things that could be used (lists of closeable components). some existing helper classes used by the various nodes could also be made closeable, for a single way to shut things down.
# I'm unsure about the failure handling here when there is a chain of plugged in things. Today failures on shutdown are logged and ignored, startup is trickier to handle, and helper classes that get started by a node may not get shut down cleanly if something goes wrong in the startup chain after it. (this is something I think I handle in HADOOP-3628)
# We could use the ping operation in HADOOP-3628, tease it out to its own interface, and then any of these plugins that implement the health check would be pulled in to the node's health check.
# If the plugins took an implementation of Service in their constructor, then the plugins would have the ability to call Configured.getConf() to get the configuration, and any other bits of the standard API. I'm not sure they would need to, not having a plugin.
# We could extend the MockService to do some plugin testing, start them with the lifecycle, roll them back etc.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675085#action_12675085 ]
Doug Cutting commented on HADOOP-5257:
--------------------------------------
> Can you pl enumerate the APIs (other than start/stop) that this interface might provide?
The generic plugin manager/factory should know nothing of the interface. So the API to get plugins might look like:
List<MyInterface> plugins = PluginFactory.getPlugins(conf, "my.config.property", MyInterface.class);
This might even be a Configuration method, e.g.:
List<MyInterface> plugins = conf.getInstances("my.config.property", MyInterface.class);
It can be written fairly straightforwardly using Configuration#getClasses() and ReflectionUtils#newInstance.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681927#action_12681927 ]
dhruba borthakur commented on HADOOP-5257:
------------------------------------------
This looks good.
Since these are kinda-APIs to namenode/datanode, they should probably reside in src/hdfs/org/apache/hadoop/hdfs/server/protocol. Also, there could be just one Interface (instead of two), both the namenode and datanode should be able to use the same interface. The Interface would have two methods, start and stop. Please let me know if this sounds ok to you.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5257:
------------------------------------
Attachment: HADOOP-5257-v3.patch
{{NamenodePlugin}} and {{DatanodePlugin}} are now interfaces, instead of abstract classes.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683826#action_12683826 ]
Hadoop QA commented on HADOOP-5257:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402576/HADOOP-5257-v7.patch
against trunk revision 756256.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/112/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/112/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/112/console
This message is automatically generated.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682476#action_12682476 ]
Carlos Valiente commented on HADOOP-5257:
-----------------------------------------
bq. Yes, that sounds right. It seems like a good idea to move it to core.
Waht about {{org.apache.hadoop.util.Plugin}}? There's already a {{MemoryCalculatorPlugin}} and a {{LinuxMemoryCalculatorPlugin}} in that package.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688845#action_12688845 ]
Carlos Valiente commented on HADOOP-5257:
-----------------------------------------
bq.1. which classloader is being used to load classes?
Classes are loaded by {{Configuration.getInstances}}, which ends up calling {{Configuration.getClassByName}}, which uses the instance field {{Configuration.classloader}}. That field is initialised by this code fragment:
{code}
private ClassLoader classLoader;
{
classLoader = Thread.currentThread().getContextClassLoader();
if (classLoader == null) {
classLoader = Configuration.class.getClassLoader();
}
}
{code}
bq.2. If parsing a string value to a list is useful, this should really go into Configuration, not the plugin classes, as that is one places to implement string trim policy, write the unit tests, etc.
I'm not sure I follow you on this point, Steve: Class name parsing is delegated to {{Configuration.getClasses}} already (which delegates the splitting to {{StringUtils.getStrings}}, it seems).
bq.3. I like the tests, add one to try loading a class that isn't there
{{org.apache.hadoop.conf.TestGetInstances}} already does that:
{code}
try {
conf.setStrings("some.classes",
SampleClass.class.getName(), AnotherClass.class.getName(),
"no.such.Class");
conf.getInstances("some.classes", SampleInterface.class);
fail("no.such.Class does not exist");
} catch (RuntimeException e) {}
{code}
Do you think it would be better to write it in a different way?
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5257) Export namenode/datanode
functionality through a pluggable RPC layer
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688829#action_12688829 ]
Steve Loughran commented on HADOOP-5257:
----------------------------------------
My other issue is that there are some very hard coded assumption that this code is limited to the filesystem. As doug says, a {{ServicePlugin}} would be more generic. You'd have to have separate classes for the {{NamenodeServicePlugin}} and the {{DatanodeServicePlugin}}, but you wouldn't need more interfaces when you came to add support for TaskTracker, JobTracker, Pig, etc.
> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
> Key: HADOOP-5257
> URL: https://issues.apache.org/jira/browse/HADOOP-5257
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch, HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
> public abstract datanodeStarted(DataNode datanode);
> public abstract datanodeStopping();
> public abstract namenodeStarted(NameNode namenode);
> public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:
> {code}public class NameNode {
> // [..]
> private void initialize(Configuration conf)
> // [...]
> for (Plugin p: PluginManager.loadPlugins(conf))
> p.namenodeStarted(this);
> }
> // [..]
> public void stop() {
> if (stopRequested)
> return;
> stopRequested = true;
> for (Plugin p: plugins)
> p.namenodeStopping();
> // [..]
> }
> // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.