You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mrql.apache.org by Apache Wiki <wi...@apache.org> on 2013/10/01 21:09:48 UTC

[Mrql Wiki] Trivial Update of "Committers" by LeonidasFegaras

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Mrql Wiki" for change notification.

The "Committers" page has been changed by LeonidasFegaras:
https://wiki.apache.org/mrql/Committers?action=diff&rev1=9&rev2=10

  handling Abstract Syntax Trees (ASTs) easier. More information can be
  found at the [[GenDoc|Gen Documentation]].  Source files ending with
  {{{.gen}}} can be be translated to plain Java files by the Java program
- {{{Gen.Main}}}, available at the {{{lib/gen.jar}}}.  You should never
+ {{{Gen.Main}}}, available at the {{{lib/mrql-gen-*.jar}}}.  You should never
  edit generated Java files; instead, you should edit the original
  {{{.gen}}} files and then convert them to Java.
  
+ == Code Organization ==
+ 
+ The Java source code is located at {{{src/main/java}}}. It contains 5 subdirectores: {{{gen}}}, {{{core}}}, {{{MapReduce}}}, {{{BSP}}}, and {{{spark}}}, for the Gen preprocessor, the core MRQL utilities used by all excution engines, the !MapReduce execution engine on Apache Hadoop, the BSP execution engine on Apache Hama, and the spark execution engine on Apache Spark.
+ 
  == The Run-Time Data ==
  
  All data used by the MRQL execution engine are encoded as instances of the '''MRData''' class.
- MRData are !WritableComparable data and are similar to [[http://avro.apache.org|AVRO]] data.
+ MRData are !WritableComparable data and are similar to the AVRO data format.
  The '''MRContainer''' class is a container for MRData that implements
  data serialization and deserialization. The following are the MRData subclasses:
  
@@ -218, +222 @@

  
    GeneratorInputFormat:: for a data generator for the  !GeneratorDataSource.
  
- These input formats are implemented differently for !MapReduce and BSP evaluation.
+ These input formats are implemented differently for !MapReduce, BSP, and Spark evaluation.
  
  === MapReduce Evaluation ===
  
- Physical plans are evaluated in !MapReduce mode by the {{{Evaluator.evalD}}} method. 
+ Physical plans are evaluated in !MapReduce mode by the {{{MapReduceEvaluator.eval}}} method. 
  
  ==== The MapReduce Plans ====
  
@@ -265, +269 @@

  
  ==== Evaluating BSP Plans on Hama ====
  
- The '''BSP/Evaluator''' class is the interpreter that executes BSP plans.
+ The '''BSPEvaluator''' class is the interpreter that executes BSP plans.
- The '''BSP/BSPPlan''' class contains the code for evaluating a BSP plan using Hama. It uses the !FileInputFormats provided by Hama, instead of those from Hadoop. Currently, messages between BSP peers are buffered into bags (which can be spilled to local files). This buffering may not be necessary with the new Hama implementations that dumps the messages to local files.
+ The '''BSPPlan''' class contains the code for evaluating a BSP plan using Apache Hama. It uses the !FileInputFormat's provided by Hama, instead of those from Hadoop. Currently, messages between BSP peers are buffered into bags (which can be spilled to local files). This buffering may not be necessary with the new Hama implementations that dumps the messages to local files.
  
  === Spark Evaluation ===
  
- The '''Spark/Evaluator''' class is the interpreter that evaluates the !MapReduce plans in Spark mode. The evaluation is straightforward because for each MRQL !MapReduce plans there is an equivalent Spark operation. The only complication is that Spark uses its own Java functions to capture anonymous functions, so MRQL '''Function''' values had to be mapped to Spark functions.
+ The '''!SparkEvaluator''' class is the interpreter that evaluates the !MapReduce plans in Spark mode. The evaluation is straightforward because for each MRQL !MapReduce plan there is an equivalent Spark operation. The only complication is that Spark uses its own Java functions to capture anonymous functions, so MRQL '''Function''' values had to be mapped to Spark functions.