You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars George (JIRA)" <ji...@apache.org> on 2012/06/26 20:35:44 UTC
[jira] [Created] (HBASE-6275) Add conditional Hadoop properties
assigment
Lars George created HBASE-6275:
----------------------------------
Summary: Add conditional Hadoop properties assigment
Key: HBASE-6275
URL: https://issues.apache.org/jira/browse/HBASE-6275
Project: HBase
Issue Type: Improvement
Components: client, master, regionserver
Affects Versions: 0.96.0
Reporter: Lars George
Priority: Minor
Fix For: 0.96.0
See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
{noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
As well as this subsequently sporting the same issue:
{noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
And the shell does:
{noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
Talking to Stack he suggest:
{quote}We should make a little function under util to do it because it will be
reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408028#comment-13408028 ]
Zhihong Ted Yu commented on HBASE-6275:
---------------------------------------
bq. do we want to support three different version of Hadoop
I think we should be prepared to.
bq. then using the oldest want work
I guess you meant "won't work"
We actually know the hadoop version HBase is using (hadoop jar file in the classpath). If we support multiple hadoop versions (1.0, 0.23 and 2.0, 3.0 to come), we can consider utilizing the effective hadoop version at runtime.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-6275:
-------------------------------
Summary: Add conditional Hadoop properties assignment (was: Add conditional Hadoop properties assigment)
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6275:
-------------------------
Status: Patch Available (was: Open)
Marking this as patch available. This would be a nice one to have. The spew in the logs makes us look messy. Needs work to finish though.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: Client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6275:
-------------------------
Tags: noob
Labels: noob (was: )
Marking noob in case some one wants to bring LarsG's work home.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: Client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407457#comment-13407457 ]
Zhihong Ted Yu commented on HBASE-6275:
---------------------------------------
I think one parameter for handleHadoopDeprecation() would be enough. handleDeprecation() would provide the new key.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405305#comment-13405305 ]
Lars George commented on HBASE-6275:
------------------------------------
We could also actually use the handleDeprecation() to replace old with new keys. But on the other hand, I think we should know where we are using outdated keys and update the code once we drop the support for the old keys. For the time being all that is needed is to avoid the warning messages. That is what this patch does.
Please have a look and let me know if this is the right approach.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465374#comment-13465374 ]
Hadoop QA commented on HBASE-6275:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534363/HBASE-6275.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2956//console
This message is automatically generated.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: Client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George reassigned HBASE-6275:
----------------------------------
Assignee: Lars George
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407979#comment-13407979 ]
Lars George commented on HBASE-6275:
------------------------------------
Hey Ted, I thought about that as well, but that has some implications, the worst is that we are using the old only in the code. Using both prepares us to use the new one once the old is completely removed. But if others think that is OK for now, then I can change it. I was tossing up either idea and went with the dual parameter approach.
This raises a larger question: how do we handle other parameters. Basically any Hadoop parameter should be wrapped into the handleHadoopDeprecation() call, so that it is handled appropriately for future versions. But if we have to handle a deprecation of an already replaced config key (does that happen?) then using the oldest want work, and the one in between only works with the previous and current version of hadoop.
Maybe we need to change the method to handleHadoopDeprecation(String[] keys) and call it like so
{code}
handleHadoopDeprecation({ oldestKey, deprecatedKey, currentKey })
{code}
The question is, do we want to support three different version of Hadoop (different as in changed config keys)?
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6275) Add conditional Hadoop properties
assignment
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-6275:
-------------------------------
Attachment: HBASE-6275.patch
The first patch attempt. I am using a helper that checks which version of Hadoop it is based on the existence of the Configuration.handleDeprecation() method.
> Add conditional Hadoop properties assignment
> --------------------------------------------
>
> Key: HBASE-6275
> URL: https://issues.apache.org/jira/browse/HBASE-6275
> Project: HBase
> Issue Type: Improvement
> Components: client, master, regionserver
> Affects Versions: 0.96.0
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6275.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-3639, we should use VersionInfo to put the proper one in, yet only one of them. Currently we always get this message when you start a daemon or the shell:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS{noformat}
> As well as this subsequently sporting the same issue:
> {noformat}2012-06-25 16:13:44,819 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id{noformat}
> And the shell does:
> {noformat}12/06/25 16:05:26 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available{noformat}
> Talking to Stack he suggest:
> {quote}We should make a little function under util to do it because it will be
> reused in a bunch of places (in daemons, shell, out in scripts, etc).{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira