You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Leon Mergen (JIRA)" <ji...@apache.org> on 2008/08/09 22:14:44 UTC
[jira] Created: (HADOOP-3932) Two small impromevents to pipes
Two small impromevents to pipes
-------------------------------
Key: HADOOP-3932
URL: https://issues.apache.org/jira/browse/HADOOP-3932
Project: Hadoop Core
Issue Type: Improvement
Components: pipes
Affects Versions: 0.17.1
Environment: n/a
Reporter: Leon Mergen
Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
- Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
- Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3932) Two small improvements to pipes
Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leon Mergen updated HADOOP-3932:
--------------------------------
Summary: Two small improvements to pipes (was: Two small impromevents to pipes)
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Attachments: pipes.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3932) Two small improvements to pipes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621691#action_12621691 ]
Owen O'Malley commented on HADOOP-3932:
---------------------------------------
Can you please regenerate the patch with svn diff as outlined in http://wiki.apache.org/hadoop/HowToContribute?
It looks like you have some whitespace changes that have sneaked into your patch. Please remove them.
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Assignee: Leon Mergen
> Attachments: pipes.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3932) Two small improvements to pipes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HADOOP-3932:
-------------------------------------
Assignee: Leon Mergen
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Assignee: Leon Mergen
> Attachments: pipes.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3932) Two small improvements to pipes
Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leon Mergen updated HADOOP-3932:
--------------------------------
Attachment: nullpointer_fix.patch
Fixes the described NullPointerException in pipes/Application.java
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Assignee: Leon Mergen
> Attachments: nullpointer_fix.patch, pipes.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3932) Two small impromevents to pipes
Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leon Mergen updated HADOOP-3932:
--------------------------------
Attachment: pipes.patch
> Two small impromevents to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Attachments: pipes.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3932) Two small improvements to pipes
Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leon Mergen updated HADOOP-3932:
--------------------------------
Attachment: pipes_interface.patch
Adds a public method in in pipes/Submitter.java that allows background submission of a job.
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Assignee: Leon Mergen
> Attachments: nullpointer_fix.patch, pipes_interface.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3932) Two small improvements to pipes
Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leon Mergen updated HADOOP-3932:
--------------------------------
Attachment: (was: pipes.patch)
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Assignee: Leon Mergen
> Attachments: nullpointer_fix.patch, pipes_interface.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3932) Two small improvements to pipes
Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622937#action_12622937 ]
Leon Mergen commented on HADOOP-3932:
-------------------------------------
Owen,
I have created two separate patches for this using svn diff.
Be aware that the second patch, "pipes_interface.patch" actually changes the interface of the Submitter.submitJob () method, and can very well cause applications to break. However, I am of the opinion that the name "submitJob ()" for a function that actually does a "runJob ()" on the JobClient was wrong to begin with.
The pipes Submitter class should conform to the mapreduce JobClient's interface naming: runJob () submits the job to the map/reduce framework and then polls for progress until the job is complete, and submitJob () should just submit the job to the map/reduce framework in the backrground.
> Two small improvements to pipes
> -------------------------------
>
> Key: HADOOP-3932
> URL: https://issues.apache.org/jira/browse/HADOOP-3932
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.17.1
> Environment: n/a
> Reporter: Leon Mergen
> Assignee: Leon Mergen
> Attachments: nullpointer_fix.patch, pipes_interface.patch
>
>
> Working with the ..mapred.pipes class a bit today, I found one bug and one possible interface improvement:
> - Application.java in its constructor assumes that DistributedCache.getLocalCacheFiles () always returns non-zero, and returns an array with at least one element -- appropriate checks and exceptions should be thrown here.
> - Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act just like the JobClient counterparts.
> Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.