You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jeff Hammerbacher (JIRA)" <ji...@apache.org> on 2010/06/09 23:41:13 UTC
[jira] Created: (MAPREDUCE-1849) Implement a FlumeJava-like library
for operations over parallel collections using Hadoop MapReduce
Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
--------------------------------------------------------------------------------------------------
Key: MAPREDUCE-1849
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Jeff Hammerbacher
The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877506#action_12877506 ]
Luke Lu commented on MAPREDUCE-1849:
------------------------------------
I had some experience with Cascading in production code. One of the major benefits of being a java library from my POV is easy unit testing of various user defined operations, which is inconvenient in most DSLs. OTOH, Cascading forces you to define data-flows explicitly (which is not so bad, if you have nice FlowBuilder utility class).
FlumeJava, IMO, actually captures the essence of MapReduce originated from functional programming. The immutable P* collections and side-effect free (no global effect) DoFn's allows many optimization opportunities a la Haskell's lazy evaluation (deferred evaluation in the paper.) However the lack of type inference and closure in Java makes the usage much more verbose than necessary. I think similar libraries could be better implemented in Scala.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877224#action_12877224 ]
Jeff Hammerbacher commented on MAPREDUCE-1849:
----------------------------------------------
h2. Data Model
* "The central class of the FlumeJava library is *PCollection<T>*, a (possibly huge) immutable bag of elements of type T."
** Can be unordered (collection) or ordered (sequence)
** Could be created with an underlying Java Collection<T> for local execution
** recordsOf() can be used to indicate how to read the elements of the collection (cf. Pig's LoadFunc or Hive's SerDe)
* Second central class: *PTable<K, V>*
** Immutable multi-map with keys of class K and values of class V
** Subclass of PCollection<Pair<K, V>>
h2. Operators
* parallelDo(PCollection<T>): PCollection<S>; runs S doFunc(T) over each element
* groupByKey(PTable<Pair<K,V>>): PTable<Pair<K, Collection<V>>>: turns a multi-map into a uni-map
* combineValues(PTable<Pair<K, Collection<V>>): PTable<Pair<K, V>>: does the reduction
* flatten(): logical view of multiple PCollections as one PCollection
* writeToRecordFileTable() to flush the output of a pipeline to a table
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877880#action_12877880 ]
Doug Cutting commented on MAPREDUCE-1849:
-----------------------------------------
I think this is better done as a separate project. Our goal in the MapReduce project should be to provide a low-level execution engine for higher-level APIs like this. Wherever possible we should strive to reduce the amount of user-level code in the base mapreduce system. This permits user-code to be versioned independently from the critical fault-tolerant base system. The base system should focus on reliability and performance, not on high-level features.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Olivier Grisel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877604#action_12877604 ]
Olivier Grisel commented on MAPREDUCE-1849:
-------------------------------------------
I am not sure closure such as scala's would work on a distributed, multi JVM setup such as Hadoop. Otherwise I agree with Luke's POV.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895314#action_12895314 ]
Ted Dunning commented on MAPREDUCE-1849:
----------------------------------------
The github implementation of Plume now supports local evaluation in an eager as opposed to lazy fashion. Avro file reading is working in at least one example. The execution plan optimizer is beginning to work. We have an emulated map-reduce framework working to support mocking up the full map-reduce execution. Nobody is working on the Hadoop interface yet so if there is a volunteer for that, they would be very welcome.
Anybody who would like to contribute is welcome as long as they are willing to Apache license their contributions.
See http://github.com/tdunning/Plume for the source, http://tdunning.blogspot.com/2010/07/new-grool.html for some discussion.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877277#action_12877277 ]
Ted Dunning commented on MAPREDUCE-1849:
----------------------------------------
Another sweet trick for this would be to allow multiple modes of execution that are all efficiently implemented. These should include local threaded, non-redundant distributed map-reduce (a la Twister) and full-on Hadoop. That gives highest speed for small jobs, medium speed for medium jobs at the cost of task failure = job resubmit and full scalability and reliability for the largest jobs.
Right now, anything but full scale hadoop execution is the red-headed child and gets no love.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877464#action_12877464 ]
Jeff Hammerbacher commented on MAPREDUCE-1849:
----------------------------------------------
Some things you get for free from being a Java library: control flow (branching, looping, etc.), composability (functions, classes, packages), IDE support, etc. Having PigLatin execute on top of something like FlumeJava could be interesting
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877435#action_12877435 ]
Owen O'Malley commented on MAPREDUCE-1849:
------------------------------------------
I haven't read the paper yet, but can you summarize how this differs from Pig? Those operators all map into Pig's operators one to one. Pig also supports join, which is *really* nice to have automated support for.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895389#action_12895389 ]
Luke Lu commented on MAPREDUCE-1849:
------------------------------------
IMO, this kind of "fluent"/LINQish interface might be better built on top of pig (preferably pig libraries, a la LINQ for SQL etc.) as you don't want/need to reinvent much of the scheduling/optimization in these existing work.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877270#action_12877270 ]
Jake Mannix commented on MAPREDUCE-1849:
----------------------------------------
+1 from this casual observer over from Mahout-land (nobody ever seems to believe me that this would make Hadoop programmers soooooo much more efficient).
I've written a half-baked, bug-ridden, inefficient version of this several times in the past, and it would be *so* useful to have done right.
An api which essentially wrapped a SequenceFile<K,V> and allowed you to do things like
Path dataPath = new Path("hdfs://foo/bar");
PTable<K,V> data = new PTable<K,V>(dataPath);
LightWeightMap<K,V,KOUT,VOUT> map = new MyMapper();
PTable<KOUT,VOUT> transformedData = data.parallelDo(map);
etc. would be awesome.
Of course, the real trick is writing a good optimizer which can figure out how to squish together separate M/R steps into one (for example, parallelDo() returns a PCollection, which you might then do groupByKey() on, but these could often easily be combined into the Map and Reduce steps of a single job).
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877451#action_12877451 ]
Jeff Hammerbacher commented on MAPREDUCE-1849:
----------------------------------------------
Owen: sure. They provide "derived operators" as well, like count(), join(), and top(). The main difference from Pig seems to be allowing users to work in Java. In fact, the Google team initially implemented their approach in a new language called Lumberjack, but mentions that, among other things, the implementation of a new language was a lot of work, and most importantly, novelty is an obstacle to adoption. They settled on Java and seem to have had some internal success.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894152#action_12894152 ]
Ted Dunning commented on MAPREDUCE-1849:
----------------------------------------
See http://tdunning.blogspot.com/2010/07/new-grool.html
I have started a github with an eager, local execution approximate clone of FlumeJava. My thought is to work out the API design before moving to parallel execution.
My current implementation has a word count example and has flushed out at least one issue in the API design (which google seems to have gotten right, btw). It could use an Avro expert to look at it to guide how to integrate Avro into the type structure.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877615#action_12877615 ]
Jakob Homan commented on MAPREDUCE-1849:
----------------------------------------
bq. I am not sure closure such as scala's would work on a distributed, multi JVM setup such as Hadoop. Otherwise I agree with Luke's POV.
Matei and the Spark guys got it working quite well: http://www.cs.berkeley.edu/~matei/spark/
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877637#action_12877637 ]
Jake Mannix commented on MAPREDUCE-1849:
----------------------------------------
While I agree that doing cool Hadoop via functional JVM languages like Scala and Clojure are great ideas, I think part of the point of the findings of this paper (and the point of this particular JIRA ticket) is concerning a simple, object-oriented Java API which has "distributed primitives" that the typical java programmer can easily understand and integrate with their current code with minimal effort.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877477#action_12877477 ]
Jake Mannix commented on MAPREDUCE-1849:
----------------------------------------
[quote]
The main difference from Pig seems to be allowing users to work in Java.
[quote]
To add my $0.02: FlumeJava lets the developers work in an object-oriented language, *period*. The difference between writing a Pig "script", or a SQL (or Hive variant therof) "query" and being able to seamlessly integrate distributed primitives (primitive not meaning java primitive, but "basic building block") in a standard java program is *amazing*
The real comparison is between FlumeJava and *Cascading*, which also lets you stay in java-land, and has a query-plan optimizer. I'm no expert in Cascading, but it seems the primitives in Cascading are "verbs" related to flows, while FlumeJava really settles on a DistributedDataSet (PCollection, for them) as the object which has methods, and can be passed to methods of other (either distributed or normal) objects. I don't know if that is clearly better, but it certainly seems more in line with the way most people program in java.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1849) Implement a FlumeJava-like
library for operations over parallel collections using Hadoop MapReduce
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877897#action_12877897 ]
Luke Lu commented on MAPREDUCE-1849:
------------------------------------
Agree that this should really be a separate project, as it's at a much higher level than mapreduce per se and it should be able to utilize features from a non-mapreduce framework, say, an alternative BSP implementation.
> Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Jeff Hammerbacher
>
> The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.