You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2009/05/14 21:14:45 UTC
[jira] Created: (HADOOP-5839) fixes to ec2 scripts to allow remote
job submission
fixes to ec2 scripts to allow remote job submission
---------------------------------------------------
Key: HADOOP-5839
URL: https://issues.apache.org/jira/browse/HADOOP-5839
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/ec2
Reporter: Joydeep Sen Sarma
i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:
http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)
#a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5839) fixes to ec2 scripts to allow
remote job submission
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710590#action_12710590 ]
Hadoop QA commented on HADOOP-5839:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12408218/5839.1.patch
against trunk revision 776148.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/354/console
This message is automatically generated.
> fixes to ec2 scripts to allow remote job submission
> ---------------------------------------------------
>
> Key: HADOOP-5839
> URL: https://issues.apache.org/jira/browse/HADOOP-5839
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 5839.1.patch
>
>
> i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5839) fixes to ec2 scripts to allow remote
job submission
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HADOOP-5839:
--------------------------------------
Attachment: hadoop-5839.2.patch
retry with diff from root. can someone review this?
> fixes to ec2 scripts to allow remote job submission
> ---------------------------------------------------
>
> Key: HADOOP-5839
> URL: https://issues.apache.org/jira/browse/HADOOP-5839
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 5839.1.patch, hadoop-5839.2.patch
>
>
> i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5839) fixes to ec2 scripts to allow remote
job submission
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HADOOP-5839:
--------------------------------------
Assignee: Joydeep Sen Sarma
Status: Patch Available (was: Open)
> fixes to ec2 scripts to allow remote job submission
> ---------------------------------------------------
>
> Key: HADOOP-5839
> URL: https://issues.apache.org/jira/browse/HADOOP-5839
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 5839.1.patch
>
>
> i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5839) fixes to ec2 scripts to allow remote
job submission
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HADOOP-5839:
--------------------------------------
Attachment: 5839.1.patch
> fixes to ec2 scripts to allow remote job submission
> ---------------------------------------------------
>
> Key: HADOOP-5839
> URL: https://issues.apache.org/jira/browse/HADOOP-5839
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Joydeep Sen Sarma
> Attachments: 5839.1.patch
>
>
> i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5839) fixes to ec2 scripts to allow remote
job submission
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-5839:
------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I've just committed this. Thanks Joydeep!
> fixes to ec2 scripts to allow remote job submission
> ---------------------------------------------------
>
> Key: HADOOP-5839
> URL: https://issues.apache.org/jira/browse/HADOOP-5839
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Fix For: 0.21.0
>
> Attachments: 5839.1.patch, hadoop-5839.2.patch
>
>
> i would very much like the option of submitting jobs from a workstation outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to internal/private IP addresses within ec2 (so traffic is optimally routed).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.