You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Tibor Kiss (JIRA)" <ji...@apache.org> on 2011/02/12 17:36:57 UTC
[jira] Issue Comment Edited: (WHIRR-168) Add a new optional c
parameter for being able to configure the port of socks connection.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993943#comment-12993943 ]
Tibor Kiss edited comment on WHIRR-168 at 2/12/11 4:36 PM:
-----------------------------------------------------------
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that functionality for client. I am right?
was (Author: tibor.kiss):
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
Can you explain me, why the mentioned clusterSpec.getConfiguration().getString("hadoop.socks.server") returns null?
I was expecting that I can query that parameter from clusterSpec.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira