You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2008/09/26 19:58:44 UTC
[jira] Created: (HADOOP-4293) Remove WritableJobConf
Remove WritableJobConf
-----------------------
Key: HADOOP-4293
URL: https://issues.apache.org/jira/browse/HADOOP-4293
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 0.19.0
The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4293) Remove WritableJobConf
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4293:
----------------------------------
Attachment: 4293-2.patch
This patch:
1. Changes the name of Configuration.write to writeXml.
2. Makes Configuration Writable.
3. Makes the new binary format a vint length, followed by 2*N Texts.
4. Removes WritableJobConf from the mapred package.
I'll create a new jira proposing that we use the binary file format instead of the job.xml for job submission.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4293) Remove WritableJobConf
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4293:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635283#action_12635283 ]
Alejandro Abdelnur commented on HADOOP-4293:
--------------------------------------------
For HADOOP-3702 we needed to serialize a {{JobConf}} into a {{String}}.
There were 3 possible alternatives considered:
# *Make {{Configuration}} or {{JobConf}} implement {{Writable}}* The problem with this option is that {{Configuration}} already as a {{write(OutputStream)}} method (that serializes to XML) and making it to implement {{Writable}} was creating compiler conflicts when used with a {{DataOutptuStream}} instance.
# *Make a {{Serialization}} implementation for {{Configuration}} (then used by its subclasses)* There was some opposition to do this.
# *Make a subclass of {{JobConf}} that implements {{Writable}}* This was the proposed alternative.
I didn't want to do Option #1 not to break backwards compatibility (quite a few places in core and contribs).
I preferred Option #2 but I did not have an objection to Option #3.
If people feel that Option #2 is more adequate I could make that change (one of the patches was doing this approach)
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635290#action_12635290 ]
Owen O'Malley commented on HADOOP-4293:
---------------------------------------
Hmm... I don't see what is wrong with option 1. It does introduce an incompatibility, but in a place where it is unlikely to break clients. The change was *not* extensive. The entire patch is 12k and there were 9 calls to the old write. And it provides the functionality in a way that fits the expected behavior.
In HADOOP-1230, we're going deprecate JobConf. So I don't think having a subclass that adds readers/writers doesn't seem right. I'm also not very happy with the write(OutputStream) versus write(DataOutput), which can easily lead to confusion.
And using the binary format instead of the xml for job submission would make a lot of sense, be much much faster, and avoid all of the quoting problems of using xml to serialize the configuration.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635177#action_12635177 ]
Hadoop QA commented on HADOOP-4293:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391085/4293-2.patch
against trunk revision 699676.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3388/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3388/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3388/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3388/console
This message is automatically generated.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4293) Remove WritableJobConf
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4293:
----------------------------------
Release Note: Made Configuration Writable and rename the old write method to writeXml.
Hadoop Flags: [Incompatible change]
Status: Patch Available (was: Open)
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635292#action_12635292 ]
Owen O'Malley commented on HADOOP-4293:
---------------------------------------
I should add that my biggest complaint is adding a new class in the mapred package that is only used by a library class. That doesn't seem like a good direction. Furthermore, the class doesn't add any new fields, just adds on another form of serialization for the preexisting class.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636034#action_12636034 ]
Hudson commented on HADOOP-4293:
--------------------------------
Integrated in Hadoop-trunk #620 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/620/])
. Make Configuration Writable and remove unreleased
WritableJobConf. Configuration.write is renamed to writeXml. (omalley)
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634956#action_12634956 ]
Arun C Murthy commented on HADOOP-4293:
---------------------------------------
+1 - not introducing a new public api is good. In future we could use this rather than the job.xml + dom-parsing for communicating between the JobClient and JobTracker.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635552#action_12635552 ]
Arun C Murthy commented on HADOOP-4293:
---------------------------------------
+1, looks good.
I agree that option 1 is the right call here.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4293) Remove WritableJobConf
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635267#action_12635267 ]
Enis Soztutar commented on HADOOP-4293:
---------------------------------------
WritableJobConf is introduced as a work around to breaking compatibility of Configuration#write() (HADOOP-3702). If we are ok to break it, I'm +1.
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293-2.patch, 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4293) Remove WritableJobConf
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4293:
----------------------------------
Attachment: 4293.patch
> Remove WritableJobConf
> -----------------------
>
> Key: HADOOP-4293
> URL: https://issues.apache.org/jira/browse/HADOOP-4293
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.19.0
>
> Attachments: 4293.patch
>
>
> The new class WritableJobConf doesn't add value over just making JobConf Writable, and I propose that we remove it before it is released.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.