You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Tibor Kiss (JIRA)" <ji...@apache.org> on 2010/12/20 18:52:07 UTC
[jira] Created: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Add a new optional c parameter for being able to configure the port of socks connection.
----------------------------------------------------------------------------------------
Key: WHIRR-168
URL: https://issues.apache.org/jira/browse/WHIRR-168
Project: Whirr
Issue Type: New Feature
Components: core, service/hadoop
Environment: ec2
Reporter: Tibor Kiss
We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
whirr.local-socks-proxy-address=localhost:6666
Instead of configuring the port, we are able to configure the address which contains the port.
(also for the sourcecode, it looks much better to not have such a hardcoded value.)
In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated WHIRR-168:
----------------------------
Fix Version/s: 0.7.0
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Fix For: 0.7.0
>
> Attachments: core-site.xml, hdfs-site.xml, integration-server-logs.tar.gz, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995538#comment-12995538 ]
Andrei Savu commented on WHIRR-168:
-----------------------------------
+1
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995950#comment-12995950 ]
Andrei Savu commented on WHIRR-168:
-----------------------------------
I 'm seeing the same failure. I suspect this is related to the following line in {{services/hadoop/.../whirr-hadoop-default.properties}}:
{code}
hadoop-client-common.hadoop.job.ugi=root,root
{code}
Hadoop integration tests are still failing but with a different error:
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 553.398 sec <<< FAILURE!
test(org.apache.whirr.service.hadoop.integration.HadoopServiceTest) Time elapsed: 113.124 sec <<< FAILURE!
junit.framework.ComparisonFailure: null expected:<a 1> but was:<null>
at junit.framework.Assert.assertEquals(Assert.java:81)
at junit.framework.Assert.assertEquals(Assert.java:87)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.test(HadoopServiceTest.java:87)
{code}
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993960#comment-12993960 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
Looking into the details of HadoopConfigurationBuilder, it would be nice to refactor the client side configuration generation in HadoopNameNodeClusterActionHandler, in way to get rid of hadoop-site.xml because the hadoop client also complains
{code}
WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
{code}
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995990#comment-12995990 ]
Tom White commented on WHIRR-168:
---------------------------------
That looks like the problem - do you want to test it works?
Thinking about this more, should we support configuration by users in standard Hadoop XML config files? Doing so would avoid problems with quoting that users might not be aware of. This could be in addition to the way we currently (so I'm not proposing we don't commit this patch). Thoughts?
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973407#action_12973407 ]
Andrei Savu commented on WHIRR-168:
-----------------------------------
+1 I have only reviewed the patch and run the unit tests.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Priority: Minor (was: Major)
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Priority: Minor
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995976#comment-12995976 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
This was the old client side properties setup
{code}
- private Properties createClientSideProperties(ClusterSpec clusterSpec,
- InetAddress namenode, InetAddress jobtracker) throws IOException {
- Properties config = new Properties();
- config.setProperty("hadoop.job.ugi", "root,root");
- config.setProperty("fs.default.name", String.format("hdfs://%s:8020/", DnsUtil.resolveAddress(namenode.getHostAddress())));
- config.setProperty("mapred.job.tracker", String.format("%s:8021", DnsUtil.resolveAddress(jobtracker.getHostAddress())));
- config.setProperty("hadoop.socks.server", "localhost:6666");
- config.setProperty("hadoop.rpc.socket.factory.class.default", "org.apache.hadoop.net.SocksSocketFactory");
- if (clusterSpec.getProvider().endsWith("ec2")) {
- config.setProperty("fs.s3.awsAccessKeyId", clusterSpec.getIdentity());
- config.setProperty("fs.s3.awsSecretAccessKey", clusterSpec.getCredential());
- config.setProperty("fs.s3n.awsAccessKeyId", clusterSpec.getIdentity());
- config.setProperty("fs.s3n.awsSecretAccessKey", clusterSpec.getCredential());
- }
{code}
and now the 3 static properties are set by the
{code}
+# Client Common
+hadoop-client-common.hadoop.job.ugi=root,root
+hadoop-client-common.hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory
+hadoop-client-common.hadoop.socks.server=localhost:6666
{code}
it generated in core-site.xml file the following
{code}
<property>
<name>hadoop.job.ugi</name>
<value>[root, root]</value>
</property>
{code}
I'm not sure about "[" "]" simbols. What do you think?
I attach the 3 generated site.xml- files.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995047#comment-12995047 ]
Tom White commented on WHIRR-168:
---------------------------------
This looks good, +1
* In HadoopConfigurationConverter#asProperties we shouldn't use Properties#put since it allows you to put an object into a Properties instance, which is unpredictable (see the javadoc for Properties).
* Why is HadoopNameNodeClusterActionHandler#getConfigDir changed to return a String rather than a File object?
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973692#action_12973692 ]
Tom White commented on WHIRR-168:
---------------------------------
Sorry, I meant WHIRR-55 in the previous comment.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Attachment: whirr-168-1.patch
I extended HadoopConfigurationBuilder to support 'hadoop-client' prefix too.
In HadoopNameNodeClusterActionHandler#createClientSideProperties then I made the changes to support core-site.xml, mapred-site.xml and hdfs-site.xml instead of the deprecated hadoop-site.xml.
In the HadoopProxy#getProxyCommand I am using the cluster.getConfiguration().getProperty("hadoop.socks.server") instead of the hardcoded one.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994027#comment-12994027 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
In whirr-hadoop-default.properties instead of hadoop-client prefix we may have the following more prefixes:
{code}
hadoop-client-common.
hadoop-client-hdfs.
hadoop-client-mapreduce.
{code}
In this way we can add extra params to each client {core|hdfs|mapred}-site.xml.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973418#action_12973418 ]
Tom White commented on WHIRR-168:
---------------------------------
It would definitely be useful to configure the port number.
Thinking ahead to HADOOP-55 (which I'm still testing), if we call this property hadoop-common.hadoop.socks.server then it will work with the way things are done there. In particular, you don't need to add the localSocksProxyAddress property to ClusterSpec, since you can just get it from ClusterSpec.getConfiguration(). Does this make sense?
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Attachment: whirr-168-2.patch
whirr-168-2.patch allow additions of hadoop-client-common, hadoop-client-hdfs and hadoop-client-mapreduce group of properties only to the client {core|hdfs|mapred]-site.xml
I didn't run the integration test for this patch, I am willing to run in case that it was reviewed and we see no further modifications necessary.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993982#comment-12993982 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
In case of hadoop, instead of having hadoop-site.xml we would need core-site.xml, mapred-site.xml and hdfs-site.xml files. As I mentioned in my last comment, the hadoop command line console is complaining about the deprecation of hadoop-site.xml.
That is why we should reuse the current HadoopConfigurationBuilder, just making some small changes, to let return Configuration also
{code}
--- a/services/hadoop/src/main/java/org/apache/whirr/service/hadoop/HadoopConfigurationBuilder.java
+++ b/services/hadoop/src/main/java/org/apache/whirr/service/hadoop/HadoopConfigurationBuilder.java
@@ -51,23 +51,44 @@ public class HadoopConfigurationBuilder {
public static Statement buildCommon(String path, ClusterSpec clusterSpec,
Cluster cluster) throws ConfigurationException, IOException {
- Configuration config = buildCommonConfiguration(clusterSpec, cluster,
- new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES));
- return HadoopConfigurationConverter.asCreateFileStatement(path, config);
+ return HadoopConfigurationConverter.asCreateFileStatement(path,
+ buildCommonConfiguration(clusterSpec, cluster));
}
public static Statement buildHdfs(String path, ClusterSpec clusterSpec,
Cluster cluster) throws ConfigurationException, IOException {
- Configuration config = buildHdfsConfiguration(clusterSpec, cluster,
- new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES));
- return HadoopConfigurationConverter.asCreateFileStatement(path, config);
+ return HadoopConfigurationConverter.asCreateFileStatement(path,
+ buildHdfsConfiguration(clusterSpec, cluster));
}
public static Statement buildMapReduce(String path, ClusterSpec clusterSpec,
Cluster cluster) throws ConfigurationException, IOException {
- Configuration config = buildMapReduceConfiguration(clusterSpec, cluster,
+ return HadoopConfigurationConverter.asCreateFileStatement(path,
+ buildMapReduceConfiguration(clusterSpec, cluster));
+ }
+
+ public static Configuration buildCommonConfiguration(ClusterSpec clusterSpec,
+ Cluster cluster) throws ConfigurationException, IOException {
+ return buildCommonConfiguration(clusterSpec, cluster,
+ new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES));
+ }
+
+ public static Configuration buildHdfsConfiguration(ClusterSpec clusterSpec,
+ Cluster cluster) throws ConfigurationException, IOException {
+ return buildHdfsConfiguration(clusterSpec, cluster,
+ new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES));
+ }
+
+ public static Configuration buildMapReduceConfiguration(ClusterSpec clusterSpec,
+ Cluster cluster) throws ConfigurationException, IOException {
+ return buildMapReduceConfiguration(clusterSpec, cluster,
+ new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES));
+ }
+
+ public static Configuration buildClientConfiguration(ClusterSpec clusterSpec,
+ Cluster cluster) throws ConfigurationException, IOException {
+ return buildClientConfiguration(clusterSpec, cluster,
new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES));
- return HadoopConfigurationConverter.asCreateFileStatement(path, config);
}
@VisibleForTesting
@@ -102,4 +123,9 @@ public class HadoopConfigurationBuilder {
return config;
}
+ @VisibleForTesting
+ static Configuration buildClientConfiguration(ClusterSpec clusterSpec,
+ Cluster cluster, Configuration defaults) throws ConfigurationException {
+ return build(clusterSpec, cluster, defaults, "hadoop-client");
+ }
}
{code}
then in HadoopNameNodeClusterActionHandler#afterConfigure we can access it
{code}
Configuration coreSiteConf = buildCommonConfiguration(clusterSpec, cluster);
Configuration hdfsSiteConf = buildHdfsConfiguration(clusterSpec, cluster);
Configuration mapredSiteConf = buildMapReduceConfiguration(clusterSpec, cluster);
Configuration clientSiteConf = buildClientConfiguration(clusterSpec, cluster);
{code}
then it clientSiteConf with coreSiteConf has to be combined into a composite configuration.
clientSiteConf is similar with HBaseClusterActionHandler#getConfiguration() composition, but in our case clientSiteConf has to be composed with coreSiteConf too and we will end up on a core-site.xml which contains everything what we have on cluster instances plus in addition what we only need to have in client side.
The question is that hdfs-site.xml and mapred-site.xml will be the same as on the cluster instances, probably with this change on the client side we will have every properties as on the cluster instances, plus some more. Is this a good approach?
Currently on the client side we have only a few values, with this approach we will increase a little bit the amount of properties. Is this a problem?
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (WHIRR-168) Add a new optional c
parameter for being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993943#comment-12993943 ]
Tibor Kiss edited comment on WHIRR-168 at 2/12/11 4:41 PM:
-----------------------------------------------------------
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that functionality for client. In other words, the current "hadoop-common" prefix from whirr-hadoop-default.properties is to be separated into a "hadoop-client"? I think so, because on the server side we don't want to add a hadoop.socks.server property.
I am right?
was (Author: tibor.kiss):
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that functionality for client. I am right?
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995982#comment-12995982 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
Maybe this is the problem.... If I would like to load that hadoop.job.ugi, the Configuration it will return an ArrayList..
for example
{code}
Configuration conf = new PropertiesConfiguration(WHIRR_HADOOP_DEFAULT_PROPERTIES);
Configuration clientConf = conf.subset("hadoop-client-common");
List<String> jobUgi = (List<String>)clientConf.getProperty("hadoop.job.ugi");
System.out.println(">" + jobUgi + "<");
{code}
Will print out
{code}
>[root, root]<
{code}
This is the problem!
Is there an escape character before ',' to not get ArrayList?
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Attachment: local-socks-proxy-address.patch
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Status: Patch Available (was: Open)
I attached a patch which adds this feature. It was tested on cloud (I am using it for myself controlling 2 clusters) and I also run the integration tests.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-168) Extend client side
configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml
instead of hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994001#comment-12994001 ]
Tibor Kiss edited comment on WHIRR-168 at 2/12/11 11:35 PM:
------------------------------------------------------------
I extended HadoopConfigurationBuilder to support 'hadoop-client' prefix too.
In HadoopNameNodeClusterActionHandler#createClientSideProperties then I made the changes to support core-site.xml, mapred-site.xml and hdfs-site.xml instead of the deprecated hadoop-site.xml.
In the HadoopProxy#getProxyCommand I am using the cluster.getConfiguration().getProperty("hadoop.socks.server") instead of the hardcoded one.
Tom, could you review the patch? Especially the configuration compositions in HadoopNameNodeClusterActionHandler#createClientSideProperties.
was (Author: tibor.kiss):
I extended HadoopConfigurationBuilder to support 'hadoop-client' prefix too.
In HadoopNameNodeClusterActionHandler#createClientSideProperties then I made the changes to support core-site.xml, mapred-site.xml and hdfs-site.xml instead of the deprecated hadoop-site.xml.
In the HadoopProxy#getProxyCommand I am using the cluster.getConfiguration().getProperty("hadoop.socks.server") instead of the hardcoded one.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996000#comment-12996000 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
I run a integration test just for apache hadoop but still fails here.
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 611.551 sec <<< FAILURE!
test(org.apache.whirr.service.hadoop.integration.HadoopServiceTest) Time elapsed: 112.188 sec <<< FAILURE!
junit.framework.ComparisonFailure: null expected:<a 1> but was:<null>
at junit.framework.Assert.assertEquals(Assert.java:81)
at junit.framework.Assert.assertEquals(Assert.java:87)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.test(HadoopServiceTest.java:87)
{code}
The nodes it was started nicely, I was logging into, checked the namenode logs and tasktracker logs.
A job it was running aparently without problems. Unfortunately this early shutdown prohibited more detailed investigation.
Temporarily I need to get rid of this early shutdown.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995987#comment-12995987 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
The fix is straingforward.
In the whirr-hadoop-default.properties we have to escape the list separator character. Like the following.
{code}
hadoop-client-common.hadoop.job.ugi=root\,root
{code}
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995926#comment-12995926 ]
Tom White commented on WHIRR-168:
---------------------------------
The code looks good, but when I tried running the Hadoop integration test I got:
{noformat}
java.io.IOException: Failed to get the current user's information.
at org.apache.hadoop.mapred.JobClient.getUGI(JobClient.java:681)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:87)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Caused by: javax.security.auth.login.LoginException: Login failed: Parameter does contain at least one user name and one group name
at org.apache.hadoop.security.UnixUserGroupInformation.readFromConf(UnixUserGroupInformation.java:219)
at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:298)
at org.apache.hadoop.mapred.JobClient.getUGI(JobClient.java:679)
{noformat}
Any ideas what this could be?
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993987#comment-12993987 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
> We should pull up getConfiguration() to ClusterActionHandlerSupport - with an extra default service Configuration that is passed by the service. However, I don't think that's needed in this patch, so we can do it elsewhere.
What is in HBaseClusterActionHandler#getConfiguration() had a different approach, not as in WHIRR-55, because in HBase there is only one configuration group, one file. In WHIRR-55 we are able to define defaults for many files. Eventually the HadoopConfigurationBuilder and HadoopConfigurationConverter has to be refactored into a Hadoop specific and the common part moved into hadoop-core module. (at least I am trying to see locally.)
(I am expecting that soon I will rename the subject of this jira issue.)
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Attachment: mapred-site.xml
hdfs-site.xml
core-site.xml
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (WHIRR-168) Add a new optional c
parameter for being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993943#comment-12993943 ]
Tibor Kiss edited comment on WHIRR-168 at 2/12/11 4:36 PM:
-----------------------------------------------------------
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that functionality for client. I am right?
was (Author: tibor.kiss):
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
Can you explain me, why the mentioned clusterSpec.getConfiguration().getString("hadoop.socks.server") returns null?
I was expecting that I can query that parameter from clusterSpec.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss reassigned WHIRR-168:
--------------------------------
Assignee: Tibor Kiss
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-168) Extend client side
configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml
instead of hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994509#comment-12994509 ]
Tibor Kiss edited comment on WHIRR-168 at 2/14/11 9:49 PM:
-----------------------------------------------------------
whirr-168-2.patch allow additions of hadoop-client-common, hadoop-client-hdfs and hadoop-client-mapreduce group of properties only to the client {core|hdfs|mapred}-site.xml
I didn't run the integration test for this patch, I am willing to run in case that it was reviewed and we see no further modifications necessary.
was (Author: tibor.kiss):
whirr-168-2.patch allow additions of hadoop-client-common, hadoop-client-hdfs and hadoop-client-mapreduce group of properties only to the client {core|hdfs|mapred]-site.xml
I didn't run the integration test for this patch, I am willing to run in case that it was reviewed and we see no further modifications necessary.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Attachment: whirr-168-3.patch
The two issues mentioned by Tom has been fixed.
The new patch is the whirr-168-3.patch file.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993943#comment-12993943 ]
Tibor Kiss commented on WHIRR-168:
----------------------------------
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties. Then I changed a line
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server", clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to hadoop-site.xml, unfortunately when the createClientSideProperties() is to be writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<< FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0 sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
Can you explain me, why the mentioned clusterSpec.getConfiguration().getString("hadoop.socks.server") returns null?
I was expecting that I can query that parameter from clusterSpec.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-168:
------------------------------
Fix Version/s: (was: 0.6.0)
Let's make this a priority for 0.7.0.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, integration-server-logs.tar.gz, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Attachment: integration-server-logs.tar.gz
this is the listing from hdfs
{code}
drwxrwxrwx - root supergroup 0 2011-02-17 19:44 /user/root
-rw-r--r-- 3 root supergroup 4 2011-02-17 19:44 /user/root/input
drwxrwxrwx - root supergroup 0 2011-02-17 19:44 /user/root/output
drwxrwxrwx - root supergroup 0 2011-02-17 19:44 /user/root/output/_logs
drwxrwxrwx - root supergroup 0 2011-02-17 19:44 /user/root/output/_logs/history
-rw-r--r-- 3 root supergroup 17117 2011-02-17 19:44 /user/root/output/_logs/history/ec2-75-101-227-207.compute-1.amazonaws.com_1297971833111_job_201102171943_0001_conf.xml
-rw-r--r-- 3 root supergroup 0 2011-02-17 19:44 /user/root/output/_logs/history/ec2-75-101-227-207.compute-1.amazonaws.com_1297971833111_job_201102171943_0001_root_NA
drwxr-xr-x - root supergroup 0 2011-02-17 19:44 /user/root/output/_temporary
drwxr-xr-x - root supergroup 0 2011-02-17 19:44 /user/root/output/_temporary/_attempt_201102171943_0001_r_000002_0
-rw-r--r-- 3 root supergroup 0 2011-02-17 19:44 /user/root/output/_temporary/_attempt_201102171943_0001_r_000002_0/part-00002
-rw-r--r-- 3 root supergroup 0 2011-02-17 19:44 /user/root/output/part-00000
-rw-r--r-- 3 root supergroup 0 2011-02-17 19:44 /user/root/output/part-00001
{code}
On the web console the job it has 10 tasks all of them terminated successfully
I also attached the integration-server-logs.tar.gz.
Somebody may help looking into these logs, because I didn't observed the problem.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: core-site.xml, hdfs-site.xml, integration-server-logs.tar.gz, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Tibor Kiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tibor Kiss updated WHIRR-168:
-----------------------------
Description:
We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
whirr.hadoop-client.hadoop.socks.server=localhost:6667
The default port will remain of course the 6666.
was:
We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
whirr.local-socks-proxy-address=localhost:6666
Instead of configuring the port, we are able to configure the address which contains the port.
(also for the sourcecode, it looks much better to not have such a hardcoded value.)
In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
Summary: Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml (was: Add a new optional c parameter for being able to configure the port of socks connection.)
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-168) Add a new optional c parameter for
being able to configure the port of socks connection.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993961#comment-12993961 ]
Tom White commented on WHIRR-168:
---------------------------------
> I think so, because on the server side we don't want to add a hadoop.socks.server property.
Yes, I think that's right (I had mistakenly thought that it was also needed server side when I made my comment above). For client side properties, we have a convention of whirr.<service-name> (see HBaseConstants for some examples), so you should follow that for the socks address.
The NPE is caused because whirr-hadoop-default.properties is not being read in the handler. HBase has some code in HBaseClusterActionHandler#getConfiguration() to combine the cluster spec with the service defaults. I think something similar would work here. In fact, this is exactly what is needed for WHIRR-222, and probably other services. We should pull up getConfiguration() to ClusterActionHandlerSupport - with an extra default service Configuration that is passed by the service. However, I don't think that's needed in this patch, so we can do it elsewhere.
> Add a new optional c parameter for being able to configure the port of socks connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded value.)
> In order to run multiple clusters you only need to override this paramter knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-168) Extend client side configurations and
support core-site.xml, mapred-site.xml and hdfs-site.xml instead of
hadoop-site.xml
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-168:
------------------------------
Fix Version/s: 0.6.0
Status: Open (was: Patch Available)
I believe it would be useful to update & fix this patch for 0.6.0. I would use it for parallel test execution by selecting a random unused port for the SOCKS proxy.
> Extend client side configurations and support core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: core-site.xml, hdfs-site.xml, integration-server-logs.tar.gz, local-socks-proxy-address.patch, mapred-site.xml, whirr-168-1.patch, whirr-168-2.patch, whirr-168-3.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed a simple mechanism to be able to parametrize this port number.
> Therefore is required to extend client side configurations, in the same way as WHIRR-55, to be configurable a 'whirr.hadoop-client.hadoop.socks.server' to something like
> whirr.hadoop-client.hadoop.socks.server=localhost:6667
> The default port will remain of course the 6666.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira