You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Klaas Bosteels (JIRA)" <ji...@apache.org> on 2008/09/29 16:45:44 UTC
[jira] Created: (HADOOP-4304) Add Dumbo to contrib
Add Dumbo to contrib
--------------------
Key: HADOOP-4304
URL: https://issues.apache.org/jira/browse/HADOOP-4304
Project: Hadoop Core
Issue Type: New Feature
Reporter: Klaas Bosteels
Priority: Minor
Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
http://static.last.fm/dumbo/dumbo-contrib.tar.gz
and more info about Dumbo can be found here:
* Basic documentation: http://github.com/klbostee/dumbo/wikis
* Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
* Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels updated HADOOP-4304:
-----------------------------------
Attachment: hadoop-4304.patch
The comments from Doug and Owen are addressed in the attached patch. An example of a Dumbo script that uses Owen's suggestion can be found here (oowordcount.py):
http://github.com/klbostee/dumbo/wikis/example-programs
BTW: The main reason why we used Streaming instead of Swig and the Pipes C++ is because the ability to run scripts locally using UNIX pipes seems to be very valuable in practice (mostly because it makes debugging very easy). Speed was not our main concern (we don't mind sacrificing some speed in favour of programmer productivity), and non-text input can be dealt with by using special InputFormats that convert to text first.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637050#action_12637050 ]
Klaas Bosteels commented on HADOOP-4304:
----------------------------------------
The failure in the core unit tests does not seem to be related to my patch (it seems to be due to HADOOP-3874 instead), so I guess this can be committed?
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4304) Add Dumbo to contrib
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reassigned HADOOP-4304:
------------------------------------
Assignee: Klaas Bosteels
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels updated HADOOP-4304:
-----------------------------------
Attachment: hadoop-4304-v2.patch
Attached an updated patch in which the java coding style is corrected and javadoc is added.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636717#action_12636717 ]
Hadoop QA commented on HADOOP-4304:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391418/hadoop-4304-v2.patch
against trunk revision 701436.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3430/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3430/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3430/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3430/console
This message is automatically generated.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels updated HADOOP-4304:
-----------------------------------
Attachment: hadoop-4304-v3.patch
We have done a few minor bugfixes and cleanups after I submitted the previous patch. Since nothing has been committed yet, I'm attaching a new patch that incorporates these changes.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels resolved HADOOP-4304.
------------------------------------
Resolution: Later
I am closing this issue (for now) because HADOOP-1722 renders it mostly obsolete. Since typed bytes provide a cleaner and more efficient alternative to the java code that is included in dumbo right now, we will be able to remove all java code from dumbo and make it a pure python module again. Consequently, future dumbo versions will be very easy to install and use, which makes adding dumbo to contrib less of requirement. Maybe we can reconsider including dumbo in contrib once it is more stable and feature complete, but right now [github|http://github.com/klbostee/dumbo] seems to be a more convenient place to develop it further...
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647468#action_12647468 ]
Owen O'Malley commented on HADOOP-4304:
---------------------------------------
You should also not assume in your example that there is a NFS file system mounted across the cluster.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635515#action_12635515 ]
Owen O'Malley commented on HADOOP-4304:
---------------------------------------
I've only looked very casually at this, but:
* I'd suggest that run should take either methods or classes for map and reduce. That would remove the need for mapconf and reduceconf parameters that are doing initialization. Basically, I'd like to see:
{code}
class Tokenizer(Mapper):
def __init__(self):
file = open("excludes.txt","r")
self.excludes = set(line.strip() for line in file)
file.close()
def map(self, key, value, context):
for word in value.split():
if not (word in self.excludes): yield word,1
class Summer(Reducer):
def reduce(self, key, values, context):
yield key,sum(values)
{code}
Of course, I'd suggest leaving your current map and reduce methods also. So that you could either do:
{code}
dumbo.run(Tokenizer, Summer, combiner=Summer);
- or -
dumbo.run(my_map, my_reduce, combiner=my_reduce);
{code}
Personally, I'd rather have you use Swig and the Pipes C++ interface rather than streaming, but I'm biased. *Smile* (Although it would give you better performance, and allow binary data to be processed.)
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels updated HADOOP-4304:
-----------------------------------
Status: In Progress (was: Patch Available)
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639650#action_12639650 ]
Hadoop QA commented on HADOOP-4304:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391891/hadoop-4304-v3.patch
against trunk revision 704599.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3461/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3461/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3461/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3461/console
This message is automatically generated.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4304:
----------------------------------
Status: Open (was: Patch Available)
I love the short word count example! Looking through the patch, there are a couple of issues:
1. The install can't require root privs.
2. It shouldn't install dumbo into python's system directory. It should instead use the distributed cache (including creating symlinks into the cwd) and use it from there.
3. ant package copies directories into the build/hadoop-0.20-dev/contrib/dumbo directory, but it doesn't seem like the right stuff. You don't need the classes or test directory. The example directory seems to be empty.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels updated HADOOP-4304:
-----------------------------------
Status: Patch Available (was: In Progress)
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639468#action_12639468 ]
Doug Cutting commented on HADOOP-4304:
--------------------------------------
You've driven into a cul de sac in our workflow. The reporter can "resume progress", but only the assignee can go from "in progress" to "patch available". Unfortunately workflows are current very hard to edit in Jira. This is fixed in the next release (http://jira.atlassian.com/browse/JRA-7661) so we should fix it easily once we upgrade.
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639759#action_12639759 ]
Klaas Bosteels commented on HADOOP-4304:
----------------------------------------
Again, the failure in the core unit tests does not seem to be related to my patch...
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639455#action_12639455 ]
Klaas Bosteels commented on HADOOP-4304:
----------------------------------------
I was expecting hudson to pick up on this and generate a QA report, but it looks like that isn't happening. Can someone tell me what I did wrong? The only available workflow action right now is "Resolve Issue", so there's no "Submit Patch" or anything like that...
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645249#action_12645249 ]
Klaas Bosteels commented on HADOOP-4304:
----------------------------------------
For people interested in Dumbo: I've started using github again as a public code repository for now, since it seems to take a while to get this committed:
http://github.com/klbostee/dumbo
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635505#action_12635505 ]
Doug Cutting commented on HADOOP-4304:
--------------------------------------
This would make a good addition to Hadoop.
Can you please:
- provide this as a patch file, attached directly to this issue.
- import src/contrib/build-contrib.xml rather than replicate it
- use src/contrib/dumbo/src/{java,python} as the source code paths
Thanks!
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Bosteels updated HADOOP-4304:
-----------------------------------
Release Note: Dumbo (a project that aims to make Hadoop Streaming as easy as possible) added to contrib.
Status: Patch Available (was: Open)
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4304) Add Dumbo to contrib
Posted by "Klaas Bosteels (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647488#action_12647488 ]
Klaas Bosteels commented on HADOOP-4304:
----------------------------------------
It actually is necessary that the Dumbo python module gets installed in the system directory of the machine from which you run your jobs, because when you run e.g.
python wordcount.py -hadoop ~hadoop/hadoop-0.17.2.1 -input brian.txt -output brian-wc -file excludes.txt
the "dumbo.run()" call gets executed on that machine, which makes Dumbo generate and excute the Streaming command
/home/hadoop/hadoop-0.17.2.1/bin/hadoop jar /home/hadoop/hadoop-0.17.2.1/contrib/streaming/hadoop-0.17.2.1-streaming.jar -input 'brian.txt' -output 'brian-wc' -file 'excludes.txt' -mapper 'python wordcount.py map 0' -reducer 'python wordcount.py red 0' -file 'wordcount.py' -file '/usr/lib/python2.4/site-packages/dumbo.py' -jobconf 'mapred.job.name=wordcount.py'
under the hood. The part
-file '/usr/lib/python2.4/site-packages/dumbo.py'
of this automatically generated command makes sure that the Dumbo module from the system dir is put in the working dir on each cluster node (i.e. it is not necessary to install the Dumbo python module in the system dir on the cluster nodes), and the extra args supplied to "python wordcount.py" in
-mapper 'python wordcount.py map 0'
-reducer 'python wordcount.py red 0'
allow Dumbo to know that it has to run the actual map or reduce instead of generating and executing a Streaming command on the nodes. Hence, we need the ability to install Dumbo in the system dir in order to make it very easy to start jobs, which is one of the main features of dumbo (some more info about this can be found at http://github.com/klbostee/dumbo/wikis/running-programs).
I also do not think that the "classes" and "test" dirs are superfluous (but I might very well be wrong). As mentioned in the description of this ticket, Dumbo also consists of some helper Java code like e.g. a special inputformat that allows to easily use sequencefiles as input for Dumbo programs (there currently is no public documentation for these features available, but we are planning to change that once this patch gets approved), and there are also some unit tests for this helper code.
Concerning the NFS mounting I'm not really sure what you mean. Maybe that "excludes.txt" has to be in the cwd? This is handled by adding the option "-file excludes.txt" to the command (like in the command above), so there is no need for any NFS mounts...
> Add Dumbo to contrib
> --------------------
>
> Key: HADOOP-4304
> URL: https://issues.apache.org/jira/browse/HADOOP-4304
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Klaas Bosteels
> Assignee: Klaas Bosteels
> Priority: Minor
> Attachments: hadoop-4304-v2.patch, hadoop-4304-v3.patch, hadoop-4304.patch
>
>
> Originally, Dumbo was a simple Python module developed at Last.fm to make writing and running Hadoop Streaming programs very easy, but now it also consists of some (up till now unreleased) helper code in Java (although it can still be used without the Java code). We propose to add Dumbo to "src/contrib" such that the Java classes get build/installed together with the rest of Hadoop, and the Python module can be installed separately at will. A tar.gz of the directory that would have to be added to "src/contrib" is available at
> http://static.last.fm/dumbo/dumbo-contrib.tar.gz
> and more info about Dumbo can be found here:
> * Basic documentation: http://github.com/klbostee/dumbo/wikis
> * Presentation at HUG (where it was first suggested to add Dumbo to contrib): http://skillsmatter.com/podcast/home/dumbo-hadoop-streaming-made-elegant-and-easy
> * Initial announcement: http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
> For some of the more advanced features of Dumbo (in particular the ones for which the Java classes are needed) there is no public documentation yet, but we could easily fill that gap by moving some of the internal Last.fm documentation to the Hadoop wiki.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.