You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2007/12/11 12:28:43 UTC
[jira] Created: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
HADOOP-2185 breaks compatibility with hadoop-0.15.0
---------------------------------------------------
Key: HADOOP-2404
URL: https://issues.apache.org/jira/browse/HADOOP-2404
Project: Hadoop
Issue Type: Bug
Components: conf
Affects Versions: 0.16.0
Reporter: Arun C Murthy
Assignee: Konstantin Shvachko
Priority: Blocker
Fix For: 0.16.0
HADOOP-2185 removed the following configuration parameters:
{noformat}
dfs.secondary.info.port
dfs.datanode.port
dfs.info.port
mapred.job.tracker.info.port
tasktracker.http.port
{noformat}
and changed the following configuration parameters:
{noformat}
dfs.secondary.info.bindAddress
dfs.datanode.bindAddress
dfs.info.bindAddress
mapred.job.tracker.info.bindAddress
mapred.task.tracker.report.bindAddress
tasktracker.http.bindAddress
{noformat}
without a backward-compatibility story.
Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552909 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> I plan to use the same static converter in HDFS and MR and call it whenever appropriate.
What happened to the idea of per-parameter methods that are back-compatible? Why wait for HADOOP-2385?
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2404:
-------------------------------------
Status: Open (was: Patch Available)
Wrong patch, Oops, sorry!
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: asyncRPC-9.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560527#action_12560527 ]
Owen O'Malley commented on HADOOP-2404:
---------------------------------------
Konst just pointed out that we want to generate a warning, so it should look like:
{code}
public int getInteger(String deprecatedName, String newName, int defaultValue);
{code}
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557109#action_12557109 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
It sounds like you are under the impression that the original patch was just about renaming a bunch of configuration parameters.
It was not. It was about prohibiting port rolling. This is a security issue.
Security changes are always like that: people are irritated when you impose more restrictions on them.
And it is an incompatible change in its nature. What is the point of restricting if you let everybody go around it?
Suppose that I left all the parameters unchanged, then people would have had hard (even harder) time understanding what is going wrong with their code.
That is why I thought it was appropriate to change names at the same time the semantics of the parameter was changing.
I thought and still think it is more fair not to provide any backward compatibility at all in this case rather than to provide partial compatibility in the form of old name recognition.
Will not repeat all the arguments again, but it looks like they convinced Hemanth, and turned Arun into willing-to-listen state.
I understand your irritation on the configuration issues, but I don't understand why blame my or equally any other patch for not dealing with them.
I do not favor the idea of creating static getters for configuration parameters in NameNode, TaskTracker etc. classes. Just commented on that in HADOOP-2385.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Attachment: ConfigConvert2.patch
Yes, the old code was setting (by mistake I am sure) "mapred.task.tracker.report.address" instead of "...bindAddress".
The former is a non-existing configuration parameter, setting of which to anything would not cause any problems since
it is not used anywhere. But I agree fixing one bug does not release you from the responsibility of fixing another one
in the same line. Here is the patch that takes care of this problem.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557455#action_12557455 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
> If you decline to fix it in a way that others approve,
I submitted two patches, and I do not see anybody except for you disapproving the last one.
I am not declining to make changes to the configuration we talked about here, just in the different jira
and because this requires more discussion while this issue is considered as a blocker by many.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2404) HADOOP-2185 breaks
compatibility with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551673 ]
shv edited comment on HADOOP-2404 at 12/13/07 5:51 PM:
-----------------------------------------------------------------------
May be we should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.
> instead of directly reading "dfs.http.bindAddress", add a package-private static method getHttpBindAddress(Configuration)
+1 I really hate that we keep using raw names rather than providing getters/setters.
This should be done for all configuration parameters imo, and probably belongs to HADOOP-2385.
was (Author: shv):
May be should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they have are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.
> instead of directly reading "dfs.http.bindAddress", add a package-private static method getHttpBindAddress(Configuration)
+1 I really hate that we keep using raw names rather than providing getters/setters/
This should be done for all configuration parameters imo, and probably belongs to HADOOP-2385.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Status: Open (was: Patch Available)
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigConvert3.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553505 ]
Raghu Angadi commented on HADOOP-2404:
--------------------------------------
Related to the above, patch for DFSck.java as part of HADOOP-2185 seems incorrect. If we specifiy "dfs.http.bindAdress" as "0.0.0.0:50070", it will try to connect to "http://0.0.0.0:50070/....".
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2404:
----------------------------------
Assignee: Owen O'Malley (was: Konstantin Shvachko)
Status: Open (was: Patch Available)
1. The method as proposed isn't general enough to be useful to more than this context.
2. It does not handle the case where the port or just the bind address are defined in the old style, but not the other.
3. It will always generate a warning about the task tracker, since the old value was set by the code.
4. The task tracker report bind address changed syntax without changing configuration names.
5. The configuration now lets users define non-zero ports for the task tracker server.
Since Konst is sick, I'll roll a new patch.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigConvert3.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550711 ]
Arun C Murthy commented on HADOOP-2404:
---------------------------------------
Konstantin, could you also please fix any documentation, if needed?
It would be great to atleast clarify the behaviour wrt port-rolling etc. (maybe cluster_setup.html)? Thanks!
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555732#action_12555732 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> Doug, you are trying to kill two birds with this one, provide compatibility and clean up the configuration mess.
No, I'm trying to provide compatibility without creating a new mess, by using this as an opportunity to decrease the mess by doing this the way it should have been done in the original patch. The original patch should have been rejected because it was not back-compatible. We're not yet under a code freeze, so we're under no pressure to create a minimally invasive patch. We should instead be trying to write exemplary code. It is not a good practice to modify the user's configuration instance. It is a good practice to handle back-compatibility for configuration options by adding accessor methods and using those wherever the option is accessed. That's what should have been done in the original patch and that's what we should do now.
We don't need a new public class called ConfigurationConverter in the util package. That's a step in the wrong direction. Back-compatibility should be localized whenever possible, not thrown into a big pot.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557369#action_12557369 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> I thought and still think it is more fair not to provide any backward compatibility at all [ ... ]
Huh? That's a change from what you stated in [#action_12550831]. No one is asking for 100% back-compatibility here, but rather for a reasonable interpretation where possible of configuration parameters that have changed. At the very least, if we can easily detect that someone is using a feature that has been incompatibly changed, we should attempt to emit a warning, and not just let things mysteriously fail, no?
> I understand your irritation on the configuration issues, but I don't understand why blame my or equally any other patch for not dealing with them.
You imply that I am asking this issue to fix a few instances of a widespread problem unrelated to the issue. That is not the case. The issue is both specific and related.
If a config parameter is only read in a single place, then no accessor method is needed. If it is simply read in multiple places, then an accessor method is nice, since it helps prevent misspellings and makes things easier if the parameter ever requires more processing, but not mandatory. Once some processing is needed for every access to a parameter then an accessor method is required, since otherwise we'd replicate non-trivial program logic. HADOOP-2185 pushed several parameters past this threshold, since back-compatibility processing is now required when these parameters are accessed, and thus accessor methods must be added.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550445 ]
Hemanth Yamijala commented on HADOOP-2404:
------------------------------------------
One of the applications that failed was Hadoop-On-Demand, which is being used internally at Yahoo. HOD brings up the DFS and MapReduce web UI on dynamically allocated cluster nodes using the older configuration items. With Hadoop trunk, these fail, and the web UI is brought up on the default nodes. HOD thus incorrectly reports the ports where the info servers are brought up.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551099 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
Dhruba, this means that there is something else running on the same port. Most likely another data-node on the default port.
As I mentioned before this patch does not reverse port rolling behavior, just provides support for the deprecated config parameters.
And "deprecated configuration parameter" message does not appear because your configuration does not contain old names.
The application you use should explicitly specify port 0 if the actual port # is meant to be arbitrary.
Doug, I agree this would be a better solution. But this will be a much bigger change now and then when we remove
deprecation. There are 5 servers involved. So at least 5 places should be patched if we go your way.
The problem is only with HOD for now as I understand. HOD does not use Configuration class, but rather
generates xml files by directly writing into them. So there is no way they can call any methods at all.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553473 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
> It's a good idea to always access parameters through accessor methods.
Nobody is arguing with that, but this is not a part of this issue.
I like Dhruba's proposal better. It will still change several files, but definitely less than 27.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2404:
-------------------------------------
Attachment: asyncRPC-9.patch
I inserted a LOG message if asyncWrite generated an exception.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: asyncRPC-9.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550717 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
Arun, There are two +1s from you on HADOOP-2185.
It also clarifies about the port rolling and etc.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553736 ]
dhruba borthakur commented on HADOOP-2404:
------------------------------------------
I like Doug's idea of cleaning up the configuration-mess, but find option 4 equally elegant to provide backward compatibility, especially because all the backward-compatibility stuff is localized at one place. This seems to be similar to the way we handle backward compatibility in disk-format, all of it localized in one method in FsEditLog.java.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552942 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
There are six parameters involved. So that's six methods at most, and they're pretty trivial methods, no more complex than the back-compatibility patch already submitted.
public static InetSocketAddress SecondaryNamenode#getInfoBindAddress(Configuration conf);
public static InetSocketAddress DataNode#getInfoBindAddress(Configuration conf) {}
...
That doesn't seem like it should take too much design effort. This issue, unlike HADOOP-2385, isn't about improved error checking, its about back-compatibility. So HADOOP-2385 might improve the error checking of these methods, but this issue can add such methods where required for back-compatibility, and change places that currently directly access the configuration to instead call these methods.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Attachment: ConfigurationConverter.patch
This patch performs configuration conversion as discussed.
I tested it with new, old, and mixed configurations. It works for me.
Could anybody please verify it on HOD.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Status: Patch Available (was: Open)
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553196 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
I thought you proposed to get rid of "directly reading the Configuration properties", all of them. Now I understand you do not.
Where do you count the six parameters?
A total of 14 parameters were affected by HADOOP-2185, 5 of them were removed, that leaves us with 9 methods.
I counted at least 12 files that directly read *.http.bindAddress parameters from the configuration.
And 27 files that directly read "fs.default.name". So the change is not hard its just massive.
Taking into account the amount of testing, which involves old, new, and mixed values of the parameters,
and that all changes are born to be removed in the next release, I am hesitant whether
"establishing the precedent of making application-specific changes in Configuration.java"
is really such a bad thing to overweight the advantage of having all changes in precisely one class.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550869 ]
dhruba borthakur commented on HADOOP-2404:
------------------------------------------
I am unable to get it to working with this patch, I see this exception on all datanodes logs:
2007-12-12 07:15:14,063 ERROR org.apache.hadoop.dfs.DataNode: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
at java.net.ServerSocket.bind(ServerSocket.java:319)
at java.net.ServerSocket.<init>(ServerSocket.java:185)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:278)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:204)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2204)
at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2148)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2169)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:2360)
No message about "deprecated configuration parameter" appear in the datanode logs.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Status: Patch Available (was: Open)
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigConvert3.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Attachment: ConfigConvert.patch
This is the implementation of variant (4).
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553436 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> that leaves us with 9 methods.
Okay, I trust your math. 9 it is.
> And 27 files that directly read "fs.default.name". So the change is not hard its just massive.
But not destabilizing. Replacing calls to conf.get("fs.default.name") with a call to a static method that does pretty much the same thing but is back-compatible seems safe enough. 0.16 is not yet frozen.
> all changes are born to be removed in the next release
That I don't see. Won't we still want all of these methods in the next release? We might add more validation to them, and add more such methods, and perhaps call them earlier, to force early validation, but we'll still want the methods, no?
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553447 ]
dhruba borthakur commented on HADOOP-2404:
------------------------------------------
I agree that it seems a little-out-of-place to put the static conversion method in the Configuration class, especially because it deals with dfs parameters only. One alternative would be to put the static conversion method in utils and then call this conversion utility from the constructors of NameNode, DataNode, JobTracker, TaskTracker and DFSClient.
This approach will not pollute the general-purpose Configuration class with application specific conversion methods while keeping the dfs/mapreduce conversion logic in one central place.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551080 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
-1 We should not fix this in Configuration.java, but rather in the classes that use the configuration. So, if you wish to do this via Configuration fixup, add it as a method in HDFS, and call that method in HDFS where it constructs its configurations.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560643#action_12560643 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
Nah, this won't work either. Sorry missed that point when we talked.
The new property actually combines values from two old names, and the old names have different value types. E.g.,
{code}
dfs.http.bindAddress = dfs.info.bindAddress + dfs.info.port
{code}
So I need to pass 2 deprecated names and one new. This does not sound very generic already, besides I will need to
specify the values of the old parameters, which makes it messy.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2404:
-------------------------------------
Status: Patch Available (was: Open)
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: asyncRPC-9.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Comment: was deleted
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552904 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
Yes, this is exactly what this patch does, including warnings.
Where do we want the configuration converter method, which class it should belong to?
I plan to use the same static converter in HDFS and MR and call it whenever appropriate.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562351#action_12562351 ]
Hadoop QA commented on HADOOP-2404:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373902/ConfigConvert3.patch
against trunk revision 614721.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1669/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1669/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1669/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1669/console
This message is automatically generated.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigConvert3.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551105 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> There are 5 servers involved.
Then that's five one-line changes. I'd rather see those than establish the precedent of making application-specific changes in Configuration.java. Each server can call the same method. When that method is removed in the next release the compiler will identify the five calls to it that must be removed.
Better yet, you could use static methods to access these parameters. So instead of directly reading dfs.http.bindAddress, add a package-private static method getHttpBindAddress(Configuration) that HDFS calls instead of directly reading the Configuration property in six different places. That method can implement back-compatibility.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553467 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
> But not destabilizing.
It is destabilizing for most of the patches currently submitted.
> That I don't see. Won't we still want all of these methods in the next release?
I am not sure we will need then in this form.
# I'd prefer to have a separate configuration class per component combining its configuration logic rather than having
static methods all around, but I'd rather continue this discussion in the other issue.
# I'd prefer to have (a) explicit and (b) deprecated methods for conversion in order to make it clear, (a) which parameters
are backward compatible now, and (b) that the feature is not going to be supported forever.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553450 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
The root bug is that these parameters are not accessed through static accessor methods. Such methods permit changing the format of parameters, renaming parameters, etc. It's a good idea to always access parameters through accessor methods. When an incompatible change to parameters is made we should add accessor methods and switch to using them uniformly.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560524#action_12560524 ]
Owen O'Malley commented on HADOOP-2404:
---------------------------------------
We've dealt with this case before in map/reduce. I propose that we add a new method in Configuration that is general:
{code}
/**
* Get an integer from a list of attribute names prefering the first.
* Should be called like:
* conf.getInteger(new String[]{"old.name", "new.name"}, 12);
*/
public int getInteger(String[] attribute names, int default);
{code}
Since this is generic code, it is reasonable to put it into config and it enables changing attribute names pretty easily. Each of the servers then has a single call to make to get the integer and in the next version they can drop the old name and move to the current getInteger.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553565 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
We are not converging here. Your arguments are mostly about how the configuration should be structured.
Doug, you are trying to kill two birds with this one:
- provide compatibility and
- clean up the configuration mess.
I prefer to keep them separate and just concentrate on the compatibility. Because as this issue reveals
common ground on the general configuration issue will be harder to achieve.
> I also have misgivings about mutating the configuration. Won't that confuse folks?
Interesting point. If somebody uses the default configuration and expects to be able to
obtain a non-existing parameter, like "dfs.info.port" from it in some legacy code it will fail.
And there isn't much you can do about it.
That is another reason for declaring this an incompatible change.
To summarize the above we have 4 proposals:
# Declare the port patch an incompatible change and ask people to change their configurations.
# Accept current conversion patch with all changes localized in one class, and revert it after release 0.16.
Although the practice of making application-specific changes in Configuration.java is a bad practice.
# Provide static configuration parameter getters (and setters?) related to each component and call
them consistently within the code.
# Create a separate class ConfigurationConverter (packaged in hadoop.util?) with a deprecated
static method for conversion, and call the method in NameNode, DataNode, JobTracker, TaskTracker,
SeondaryNamenode, and DFSClient constructors.
I am in favor of 1, but I am ok with 2 and 4.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557441#action_12557441 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> "some processing" of exactly these parameters was introduced in HADOOP-1085. I opposed it then. You just committed it.
That looks like I made a mistake. Mea culpa. I don't recall the details, but in those days I was doing a lot of commits and my reviews may have sufferered.
> But I do not agree that they should be introdueced in this patch, which will lead to massive changes
I disagree that the changes are massive. They're easy to locate (points where the modified parameters are accessed) not that many locations, and only affect a line or two of code at each location. I also disagree that the size alone of the change should be a significant factor here. The change is simple enough that it will not be destabilizing. The places changed are not likely to be touched by many other pending patches, so it should not create many conflicts.
> This argument is going on for almost a month now. I do not find it productive.
> I mean, people can have different opinions, what do you do with that.
If committers cannot reach consensus, then the issue can be taken to the PMC, although that seems like overkill in this case. If you decline to fix it in a way that others approve, and it is a blocker, then someone else must develop a patch that we can all agree on before we can make the release. I think you are the best qualified person to fix this. I could try to generate a patch, but it would probably take me a lot longer than it would you and I would be more likely to make subtle errors, since I am less intimate with the changes.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2404:
-------------------------------------
bq. May be we should just declare that this change is incompatible and not provide any conversion of the changed parameters at all. We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them to be rolled if they are busy, would have to explicitly specify 0-ports. This means the configuration should be changed, so users might change the names as well.
I feel this is a fair argument. We are OK to make this change in our application.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553491 ]
Raghu Angadi commented on HADOOP-2404:
--------------------------------------
> And 27 files that directly read "fs.default.name".
I thought meaning of "fs.default.name" hasn't changed.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551673 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
May be should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they have are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.
> instead of directly reading "dfs.http.bindAddress", add a package-private static method getHttpBindAddress(Configuration)
+1 I really hate that we keep using raw names rather than providing getters/setters/
This should be done for all configuration parameters imo, and probably belongs to HADOOP-2385.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2404:
--------------------------------
Status: Open (was: Patch Available)
There is one problem that HADOOP-2185 introduced (without knowledge I am sure) in the tasktracker's reinitialization (after the jobtracker declares the tasktracker as lost, and upon the next connection to the jobtracker, the tasktracker is asked to reinitialize). Post HADOOP-2185, the tasktracker's initialize method sets "mapred.task.tracker.report.bindAddress" to the listener address. This would be in the format _/<name>:<port>_ (due to the way InetAddress.toString() is implemented). Now if the tasktracker were lost and were to reinitialize, it would call the initialize method again, and this time the call to createSocketAddress would fail because of the leading slash in the argument, and the tasktracker would not be able to recover from this. So maybe, we should do a check for '/' in the tasktracker initialization before passing it to createSocketAddress? Not sure whether such problems exist elsewhere in the framework code too ....
Pre-HADOOP-2185, the "mapred.task.tracker.report.bindAddress" would not be reset to the listener address, and, hence the above problem is not there..
This issue needs a resolution (and hence I am cancelling the patch).
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553484 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> I'd prefer to have a separate configuration class per component combining its configuration logic rather than having static methods all around, but I'd rather continue this discussion in the other issue.
I agree that configuration should be per-component. But in this case, each component already has a well-known primary class: Namenode, Datanode, JobTracker, TaskTracker, etc. Or would you rather partition things into simply mapred and hdfs? Personally, I don't think of those as components so much as sub-systems, and think that those existing classes are appropriate for these accessors long-term. If you don't agree, then I guess we'll have to push this to another issue.
I also have misgivings about mutating the configuration. That can make things trickier to debug, since the values in the configuration that are displayed won't match those that the user passed. Won't that confuse folks?
It would indeed be nice to have the back-compatibility code in a single place that can easily be deprecated, but we could instead add a consistent comment to each of these places, and file an issue now to remove all places that are so-commented.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557433#action_12557433 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
> That's a change from what you stated in action_12550831.
I don't see any contradiction. Besides I already said that many times after that, if you read a little bit further down
[#action_12551673] and [#action_12553565].
Providing partial conversion is a compromise, which I am willing to accept.
This is temporary for one release, that is why all related code is deprecated in my patch.
> You imply that I am asking this issue to fix a few instances of a widespread problem unrelated to the issue.
"some processing" of exactly these parameters was introduced in HADOOP-1085. [I opposed it then.|https://issues.apache.org/jira/browse/HADOOP-1085?focusedCommentId=12483800#action_12483800] You just committed it.
If dozens of files is not widespread then what is.
The only thing I agree with you upon is that configuration needs accessors.
But I do not agree that they should be introdueced in this patch, which will lead to massive changes,
and I do not support *static* accessors, and I do not see any of these is supported by anybody else.
This argument is going on for almost a month now. I do not find it productive.
Unless the compromise proposed by Dhruba is acceptable for you,
I am planning to submit the bug fix mentioned by Devaraj in a separate issue, and let somebody else deal with this one.
I mean, people can have different opinions, what do you do with that.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561437#action_12561437 ]
Owen O'Malley commented on HADOOP-2404:
---------------------------------------
Then please make a getter in the NameNode to pull this value out. The ones that are just attribute renamings can be done the other way.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2404:
-------------------------------------
Attachment: (was: asyncRPC-9.patch)
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562770#action_12562770 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
> 3. It will always generate a warning about the task tracker, since the old value was set by the code.
This is not true. It will not generate warning because the new code sets the old value as well.
> Since Konst is sick, I'll roll a new patch.
Yes, please.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigConvert3.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552832 ]
Arun C Murthy commented on HADOOP-2404:
---------------------------------------
{quote}
May be we should just declare that this change is incompatible and not provide any conversion of the changed parameters at all.
We cannot provide true compatibility. Whoever relied on the default port usage expecting (consciously or unconsciously) them
to be rolled if they are busy, would have to explicitly specify 0-ports. This means the configuration should be changed,
so users might change the names as well.
{quote}
I'm *ok* with some pieces changing semantics, but I worry about current configs breaking - to be honest most people don't care as much about the port-rolling semantics as they do about having their current hadoop-site.xml work as-is. Given that I believe it's important to maintain compatibility... however I'm willing to listen.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550831 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
HADOOP-2185 introduced two types of changes:
# Change in semantics: no port rolling for any ports.
# Configuration api changes: 5 properties have been removed and 5 renamed.
Here is the compatibility plan.
# Semantical changes will not be backward compatible.
# Old configuration variables will be recognized in 0.16 as described below.
If a new name of a configuration parameter is present, then old names corresponding
to this parameter will be ignored.
If old names of a configuration parameter are present but not the new ones
the old parameters will be converted to the new according to the conversion
table provided in HADOOP-2185.
The conversion will be done right after loading all Configuration resources
(see Configuration.loadResources()) and Hadoop will further work with the
converted configuration.
I was assured this will work fine for HOD and Pig.
But if your application uses tricks like:
create config; call new JobTracker(config); then get the job-tracker info port
by calling config.get("mapred.job.tracker.info.port")
introduced by HADOOP-1085 this will not work. You will need to change your application
to use config.get("mapred.job.tracker.http.bindAddress") and then extract the port.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552834 ]
Doug Cutting commented on HADOOP-2404:
--------------------------------------
> I worry about current configs breaking [ ... ]
Right. If, for example, a site currently sets dfs.info.port, then they should be able to upgrade and still use the web ui at that port (assuming it's not busy) without changing their configuration. They should also get some warnings on startup, encouraging them to switch to dfs.info.bindAddress.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Status: Patch Available (was: Open)
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2404) HADOOP-2185 breaks compatibility with
hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-2404:
----------------------------------------
Attachment: ConfigConvert3.patch
This is in line with Owen's initial proposal.
I created a new method that merges deprecated properties into one using specified delimiter.
{code}
public String get(String[] deprecatedNames,
String delimiter,
String newName,
String defaultValue) {
{code}
For the purpose of this issue I always pass two deprecated names that correspond to an address and a port of each component.
The method can be also used for simple cases, like rename of a property.
In this case the array should contain only one entry.
The types are always treated as Strings, which works well for xml properties.
Type verification will be done later when the value is converted to its actual type.
Also, I commented all deprecated properties in the code so that we could easily remove them later.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigConvert.patch, ConfigConvert2.patch, ConfigConvert3.patch, ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2404) HADOOP-2185 breaks compatibility
with hadoop-0.15.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552940 ]
Konstantin Shvachko commented on HADOOP-2404:
---------------------------------------------
Do you propose to introduce here per-parameter (back-compatible) methods only for
the affected address and port parameters or for all of them?
The latter is a big change presumably to be addressed by HADOOP-2385 after a reasonable design effort.
Backward compatibility for port parameters is a more urgent and much less permanent matter.
> HADOOP-2185 breaks compatibility with hadoop-0.15.0
> ---------------------------------------------------
>
> Key: HADOOP-2404
> URL: https://issues.apache.org/jira/browse/HADOOP-2404
> Project: Hadoop
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.0
> Reporter: Arun C Murthy
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: ConfigurationConverter.patch
>
>
> HADOOP-2185 removed the following configuration parameters:
> {noformat}
> dfs.secondary.info.port
> dfs.datanode.port
> dfs.info.port
> mapred.job.tracker.info.port
> tasktracker.http.port
> {noformat}
> and changed the following configuration parameters:
> {noformat}
> dfs.secondary.info.bindAddress
> dfs.datanode.bindAddress
> dfs.info.bindAddress
> mapred.job.tracker.info.bindAddress
> mapred.task.tracker.report.bindAddress
> tasktracker.http.bindAddress
> {noformat}
> without a backward-compatibility story.
> Lots are applications/cluster-configurations are prone to fail hence, we need a way to keep things working as-is for 0.16.0 and remove them for 0.17.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.