You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Thomas Garner (JIRA)" <ji...@apache.org> on 2008/01/11 08:07:34 UTC
[jira] Created: (HADOOP-2578) HBaseConfiguration(Configuration c)
constructor shouldn't overwrite passed conf with defaults
HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed conf with defaults
---------------------------------------------------------------------------------------------
Key: HADOOP-2578
URL: https://issues.apache.org/jira/browse/HADOOP-2578
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Reporter: Thomas Garner
While testing out mapreduce with hbase, in the map portion of a task, the map would try to connect to an hbase master at localhost/127.0.0.1. The config passed to the hbaseconfiguration contained the necessary hbase configuration information, but I assume was being overwritten by the defaults in the config files during addResource, as commenting out addHbaseResources in the constructor fixed the symptom. I would expect the configs to be layered on top of each other, e.g. default, then site, then the config passed as a parameter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2578) HBaseConfiguration(Configuration c)
constructor shouldn't overwrite passed conf with defaults
Posted by "Thomas Garner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558286#action_12558286 ]
Thomas Garner commented on HADOOP-2578:
---------------------------------------
I'm not sure I understand your logic. Are you saying that when hbase is reading hbase-default and hbase-site, that it's not finding them on disk (they are there and properly configured in /usr/local/hadoop/src/contrib/hbase/conf on both of my mapreduce nodes), and then injecting defaults from somewhere other than the xml files?
To test this, I edited hbase-default on one node, and I am now seeing the change (hbase.master to default:60000):
2008-01-12 19:56:59,675 DEBUG org.apache.hadoop.mapred.TaskTracker: Child starting
2008-01-12 19:57:00,038 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2008-01-12 19:57:00,284 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2008-01-12 19:57:00,353 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after super(c): hbase.master: master:60000
2008-01-12 19:57:00,420 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after hbase-default.xml: hbase.master: default:60000
2008-01-12 19:57:00,420 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after hbase-site.xml: hbase.master: default:60000
2008-01-12 19:57:00,428 DEBUG org.apache.hadoop.hbase.HConnectionManager: DEBUG: HConnection getConnection(): hbase.master: default:60000
2008-01-12 19:57:00,438 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after super(c): hbase.master: default:60000
2008-01-12 19:57:00,444 DEBUG org.apache.hadoop.hbase.HConnectionManager$TableServers: No servers for wordcount. Doing a find...
2008-01-12 19:57:10,119 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 0 of 5 failed with . Retrying after sleep of 10000
2008-01-12 19:57:20,132 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 1 of 5 failed with . Retrying after sleep of 10000
2008-01-12 19:57:30,144 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 2 of 5 failed with . Retrying after sleep of 10000
2008-01-12 19:57:40,153 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 3 of 5 failed with . Retrying after sleep of 10000
Here's the log from an earlier mr job/task, where you see the original hbase.master value be correct (master:60000), then change to local (as was the value in the hbase-default file):
2008-01-12 19:16:49,005 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2008-01-12 19:16:49,253 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2008-01-12 19:16:49,333 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after super(c): hbase.master: master:60000
2008-01-12 19:16:49,403 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after hbase-default.xml: hbase.master: local
2008-01-12 19:16:49,404 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after hbase-site.xml: hbase.master: local
2008-01-12 19:16:49,421 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after super(c): hbase.master: local
2008-01-12 19:16:49,453 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 1 time(s).
2008-01-12 19:16:50,458 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 2 time(s).
2008-01-12 19:16:51,462 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 3 time(s).
2008-01-12 19:16:52,466 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 4 time(s).
I don't have an issue allowing mr nodes to read hbase default and site configs (say for optimizations applicable to all hbase mr jobs that are pushed to disk for every node), so long as config values specific to my job override any default values.
I'm surprised that there's no set method in the configuration class that takes a configuration as a parameter, then all you'd have to do is super on the default, then set the site and set the passed config, each overriding any existing parameters.
> HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed conf with defaults
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-2578
> URL: https://issues.apache.org/jira/browse/HADOOP-2578
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: Thomas Garner
>
> While testing out mapreduce with hbase, in the map portion of a task, the map would try to connect to an hbase master at localhost/127.0.0.1. The config passed to the hbaseconfiguration contained the necessary hbase configuration information, but I assume was being overwritten by the defaults in the config files during addResource, as commenting out addHbaseResources in the constructor fixed the symptom. I would expect the configs to be layered on top of each other, e.g. default, then site, then the config passed as a parameter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2578) HBaseConfiguration(Configuration c)
constructor shouldn't overwrite passed conf with defaults
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558167#action_12558167 ]
stack commented on HADOOP-2578:
-------------------------------
Would adding a test that checked that c.getResource("hbase-default.xml") and c.getResource("hbase-site.xml") were non-null in HBaseConfiguration(Configuration c) work?
> HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed conf with defaults
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-2578
> URL: https://issues.apache.org/jira/browse/HADOOP-2578
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: Thomas Garner
>
> While testing out mapreduce with hbase, in the map portion of a task, the map would try to connect to an hbase master at localhost/127.0.0.1. The config passed to the hbaseconfiguration contained the necessary hbase configuration information, but I assume was being overwritten by the defaults in the config files during addResource, as commenting out addHbaseResources in the constructor fixed the symptom. I would expect the configs to be layered on top of each other, e.g. default, then site, then the config passed as a parameter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2578) HBaseConfiguration(Configuration c)
constructor shouldn't overwrite passed conf with defaults
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558292#action_12558292 ]
stack commented on HADOOP-2578:
-------------------------------
Pardon me. My logic was, if the hbase-*.xml files have not been added to the config. already, then go ahead and add them. I wasn't sure if it would work (in fact, thinking on it, the config. by the time it gets out to the map task, resource files will already have been read so my suggested test will likely fail). I was asking you to try it/give an opinion.
Do your cluster members have an hbase-site.xml with customizations such as hbase.master=master:60000 out on the cluster available off the CLASSPATH -- i.e. in $HADOOP_HOME/conf, or in $HADOOOP_CLASSPATH or in the job jar -- or were the hbase customizations only present on the master or added when you set up the job?
This issue has burned a few other folks in the past. Thanks for help w/ it. Need to fix it.
Related, I've been trying to keep up a page on MR+hbase here: http://wiki.apache.org/lucene-hadoop/Hbase/MapReduce. You might have some edits/additions to make.
> HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed conf with defaults
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-2578
> URL: https://issues.apache.org/jira/browse/HADOOP-2578
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: Thomas Garner
>
> While testing out mapreduce with hbase, in the map portion of a task, the map would try to connect to an hbase master at localhost/127.0.0.1. The config passed to the hbaseconfiguration contained the necessary hbase configuration information, but I assume was being overwritten by the defaults in the config files during addResource, as commenting out addHbaseResources in the constructor fixed the symptom. I would expect the configs to be layered on top of each other, e.g. default, then site, then the config passed as a parameter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.