You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by ol...@apache.org on 2009/12/12 01:35:51 UTC
svn commit: r889870 - in /hadoop/pig/trunk: ./
src/docs/src/documentation/content/xdocs/
src/docs/src/documentation/content/xdocs/images/
Author: olga
Date: Sat Dec 12 00:35:50 2009
New Revision: 889870
URL: http://svn.apache.org/viewvc?rev=889870&view=rev
Log:
PIG-1147: Zebra Docs for Pig 0.6.0 (chandec via olgan)
Added:
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/images/zebra.jpg (with props)
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_mapreduce.xml
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_overview.xml
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_pig.xml
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_reference.xml
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_stream.xml
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_users.xml
Modified:
hadoop/pig/trunk/CHANGES.txt
hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/site.xml
Modified: hadoop/pig/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/CHANGES.txt?rev=889870&r1=889869&r2=889870&view=diff
==============================================================================
--- hadoop/pig/trunk/CHANGES.txt (original)
+++ hadoop/pig/trunk/CHANGES.txt Sat Dec 12 00:35:50 2009
@@ -24,6 +24,8 @@
IMPROVEMENTS
+PIG-1147: Zebra Docs for Pig 0.6.0 (chandec via olgan)
+
PIG-1129: Pig UDF doc: fieldsToRead function (chandec via olgan)
PIG-978: MQ docs update (chandec via olgan)
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/images/zebra.jpg
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/images/zebra.jpg?rev=889870&view=auto
==============================================================================
Binary file - no diff available.
Propchange: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/images/zebra.jpg
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/site.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=889870&r1=889869&r2=889870&view=diff
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/site.xml (original)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/site.xml Sat Dec 12 00:35:50 2009
@@ -50,6 +50,14 @@
<cookbook label="Cookbook" href="cookbook.html" />
<udf label="UDFs" href="udf.html" />
</docs>
+ <docs label="Zebra">
+ <piglatin label="Zebra Overview " href="zebra_overview.html" />
+ <piglatin label="Zebra Users " href="zebra_users.html" />
+ <piglatin label="Zebra Reference " href="zebra_reference.html" />
+ <piglatin label="Zebra MapReduce " href="zebra_mapreduce.html" />
+ <piglatin label="Zebra Pig " href="zebra_pig.html" />
+ <piglatin label="Zebra Streaming " href="zebra_stream.html" />
+ </docs>
<docs label="Miscellaneous">
<api label="API Docs" href="ext:api"/>
<wiki label="Wiki" href="ext:wiki" />
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_mapreduce.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_mapreduce.xml?rev=889870&view=auto
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_mapreduce.xml (added)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_mapreduce.xml Sat Dec 12 00:35:50 2009
@@ -0,0 +1,606 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!-- Copyright 2002-2004 The Apache Software Foundation
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+ "http://forrest.apache.org/dtd/document-v20.dtd">
+
+ <!-- BEGIN DOCUMENT-->
+
+<document>
+<header>
+<title>Zebra and MapReduce</title>
+</header>
+<body>
+
+
+ <!-- OVERVIEW-->
+ <section>
+ <title>Overview</title>
+ <p>MapReduce allows you to take full advantage of Zebra's capabilities.</p>
+ </section>
+ <!-- END OVERVIEW-->
+
+<!-- HADOOP M/R API-->
+ <section>
+ <title>Hadoop MapReduce APIs</title>
+ <p>Zebra requires Hadoop 20. However, this release of Zebra only supports the "old" jobconf-style MapReduce APIs. </p>
+ <ul>
+ <li>"old" mapreduce API - org.apache.hadoop.mapred.* - supported</li>
+ <li>"new" mapreduce API - org.apache.hadoop.mapreduce.* - not supported</li>
+ </ul>
+ <p></p>
+ </section>
+<!-- END HADOOP M/R API-->
+
+
+ <!-- ZEBRA M/R EXAMPLES-->
+<section>
+<title>Zebra MapReduce Examples</title>
+
+
+
+
+ <!-- ZEBRA OUTPUT EXAMPLE-->
+<section>
+<title>Table Output Format</title>
+<p>This MapReduce example demonstrates the Zebra table output format.
+The Zebra table in this example has two unsorted columns groups, each of which has one column.
+The output format is specified as follows:</p>
+
+<source>
+BasicTableOutputFormat.setStorageInfo(jobConf,
+ ZebraSchema.createZebraSchema("word:string, count:int"),
+ ZebraStorageHint.createZebraStorageHint("[word];[count]"),
+ null);
+</source>
+
+<p><br></br>The input file for this example should contain rows of word and count, separated by a space. For example:</p>
+
+<source>
+this 2
+is 1
+a 5
+test 2
+hello 1
+world 3
+</source>
+
+<p>
+The example works like this. The first job is in Zebra format. The second job reads output from the first job, where Count is specified as a projection column.
+The table input format projects an input row which has both Word and Count into a row containing only the Count column and hands it to map.
+The reducer sums the counts and produces a sum of counts which should match total number of words in original text.
+</p>
+
+<source>
+package org.apache.hadoop.zebra.mapred;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.*;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.pig.data.Tuple;
+import org.apache.hadoop.zebra.parser.ParseException;
+import org.apache.hadoop.zebra.schema.Schema;
+import org.apache.hadoop.zebra.types.TypesUtils;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+public class TableMapReduceExample extends Configured implements Tool {
+
+ static class Map extends MapReduceBase implements Mapper<LongWritable, Text, BytesWritable, Tuple> {
+ private BytesWritable bytesKey;
+ private Tuple tupleRow;
+
+ /**
+ * Map method for reading input.
+ */
+ @Override
+ public void map(LongWritable key, Text value,
+ OutputCollector<BytesWritable, Tuple> output, Reporter reporter)
+ throws IOException {
+
+ // value should contain "word count"
+ String[] wordCount = value.toString().split(" ");
+ if (wordCount.length != 2) {
+ // LOG the error
+ throw new IOException("Value does not contain two fields:" + value);
+ }
+
+ byte[] word = wordCount[0].getBytes();
+ bytesKey.set(word, 0, word.length);
+ tupleRow.set(0, new String(word));
+ tupleRow.set(1, Integer.parseInt(wordCount[1]));
+
+ output.collect(bytesKey, tupleRow);
+ }
+
+ /**
+ * Configuration of the job. Here we create an empty Tuple Row.
+ */
+ @Override
+ public void configure(JobConf job) {
+ bytesKey = new BytesWritable();
+ try {
+ Schema outSchema = BasicTableOutputFormat.getSchema(job);
+ tupleRow = TypesUtils.createTuple(outSchema);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ } catch (ParseException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ static class ProjectionMap extends MapReduceBase implements Mapper<BytesWritable, Tuple, Text, IntWritable> {
+ private final static Text all = new Text("All");
+
+ /**
+ * Map method which gets count column after projection.
+ *
+ * @throws IOException
+ */
+ @Override
+ public void map(BytesWritable key, Tuple value,
+ OutputCollector<Text, IntWritable> output, Reporter reporter)
+ throws IOException {
+ output.collect(all, new IntWritable((Integer) value.get(0)));
+ }
+ }
+
+ public static class ProjectionReduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
+ /**
+ * Reduce method which implements summation. Acts as both reducer and combiner.
+ *
+ * @throws IOException
+ */
+ public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
+ int sum = 0;
+ while (values.hasNext()) {
+ sum += values.next().get();
+ }
+ output.collect(key, new IntWritable(sum));
+ }
+ }
+
+ /**
+ * Where jobs and their settings and sequence is set.
+ *
+ * @param args arguments with exception of Tools understandable ones.
+ */
+ public int run(String[] args) throws Exception {
+ if (args == null || args.length != 3) {
+ System.out.println("usage: TableMapReduceExample input_path_for_text_file output_path_for_table output_path_for_text_file");
+ System.exit(-1);
+ }
+
+ /*
+ First MR Job creating a Table with two columns
+ */
+ JobConf jobConf = new JobConf();
+ jobConf.setJobName("TableMapReduceExample");
+ jobConf.set("table.output.tfile.compression", "none");
+
+ // Input settings
+ jobConf.setInputFormat(TextInputFormat.class);
+ jobConf.setMapperClass(Map.class);
+ FileInputFormat.setInputPaths(jobConf, new Path(args[0]));
+
+ // Output settings
+ jobConf.setOutputFormat(BasicTableOutputFormat.class);
+ BasicTableOutputFormat.setOutputPath(jobConf, new Path(args[1]));
+
+ // set the output info:
+ // ZebraSchema - 2 logical columns
+ // ZebraStorageHint - 2 physical column groups (one column each)
+ // ZebraSortInfo - unsorted table (null)
+
+ BasicTableOutputFormat.setStorageInfo(jobConf,
+ ZebraSchema.createZebraSchema("word:string, count:int"),
+ ZebraStorageHint.createZebraStorageHint("[word];[count]"),
+ null);
+
+ // set map-only job.
+ jobConf.setNumReduceTasks(0);
+
+ // Run Job
+ JobClient.runJob(jobConf);
+
+ /*
+ Second MR Job for Table Projection of count column
+ */
+ JobConf projectionJobConf = new JobConf();
+ projectionJobConf.setJobName("TableProjectionMapReduceExample");
+
+ // Input settings
+ projectionJobConf.setMapperClass(ProjectionMap.class);
+ projectionJobConf.setInputFormat(TableInputFormat.class);
+ TableInputFormat.setProjection(projectionJobConf, "count");
+ TableInputFormat.setInputPaths(projectionJobConf, new Path(args[1]));
+ projectionJobConf.setMapOutputKeyClass(Text.class);
+ projectionJobConf.setMapOutputValueClass(IntWritable.class);
+
+ // Output settings
+ projectionJobConf.setOutputFormat(TextOutputFormat.class);
+ FileOutputFormat.setOutputPath(projectionJobConf, new Path(args[2]));
+ projectionJobConf.setReducerClass(ProjectionReduce.class);
+ projectionJobConf.setCombinerClass(ProjectionReduce.class);
+
+ // Run Job
+ JobClient.runJob(projectionJobConf);
+
+ return 0;
+ }
+
+ public static void main(String[] args) throws Exception {
+ int res = ToolRunner.run(new Configuration(), new TableMapReduceExample(), args);
+ System.exit(res);
+ }
+}
+</source>
+</section>
+ <!-- END ZEBRA TABLE OUTPUT EXAMPLE-->
+
+
+ <!-- ZEBRA OUTPUT EXAMPLE-->
+<section>
+<title>Table Input/Output Formats</title>
+<p>This MapReduce example demonstrates the Zebra table input/output formats. </p>
+ <source>
+ package org.apache.hadoop.zebra.mapred;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.ArrayList;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.mapred.JobClient;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Mapper;
+import org.apache.hadoop.mapred.OutputCollector;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.zebra.mapred.BasicTableOutputFormat;
+import org.apache.hadoop.zebra.mapred.TableInputFormat;
+import org.apache.hadoop.zebra.parser.ParseException;
+import org.apache.hadoop.zebra.schema.Schema;
+import org.apache.hadoop.zebra.types.TypesUtils;
+import org.apache.pig.data.Tuple;
+
+/**
+ * This is a sample to show using zebra table to do a simple basic union in
+ * map/reduce * To run this, we need have two basic tables ready. They contain
+ * the data as in Sample 1, i.e., (word, count). In this example, they are at:
+ * /user/mapredu/t1 /user/mapredu/t2 The resulting table is put at: /user/mapredu2/t1
+ *
+ */
+public class TableMRSample2 {
+ static class MapClass implements
+ Mapper<BytesWritable, Tuple, BytesWritable, Tuple> {
+ private BytesWritable bytesKey;
+ private Tuple tupleRow;
+
+ @Override
+ public void map(BytesWritable key, Tuple value,
+ OutputCollector<BytesWritable, Tuple> output, Reporter reporter)
+ throws IOException
+
+ {
+ System.out.println(key.toString() + value.toString());
+ output.collect(key, value);
+ }
+
+ @Override
+ public void configure(JobConf job) {
+ bytesKey = new BytesWritable();
+ try {
+ Schema outSchema = BasicTableOutputFormat.getSchema(job);
+ tupleRow = TypesUtils.createTuple(outSchema);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ } catch (ParseException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ // no-op
+ }
+
+ public static void main(String[] args) throws ParseException, IOException {
+ JobConf jobConf = new JobConf();
+ jobConf.setJobName("tableMRSample");
+ jobConf.set("table.output.tfile.compression", "gz");
+
+ // input settings
+ jobConf.setInputFormat(TableInputFormat.class);
+ jobConf.setOutputFormat(BasicTableOutputFormat.class);
+ jobConf.setMapperClass(TableMRSample2.MapClass.class);
+
+ List
+ <Path> paths = new ArrayList<Path>(2);
+ Path p = new Path("/user/mapredu/t1");
+ System.out.println("path = " + p);
+ paths.add(p);
+ p = new Path("/user/mapredu/t2");
+ paths.add(p);
+
+ TableInputFormat.setInputPaths(jobConf, paths.toArray(new Path[2]));
+ ZebraProjection zebraProjection = ZebraProjection.createZebraProjection("word");
+ TableInputFormat.setProjection(jobConf, zebraProjection);
+ BasicTableOutputFormat.setOutputPath(jobConf, new Path("/user/mapredu2/t1"));
+
+ ZebraSchema zebraSchema = ZebraSchema.createZebraSchema("word:string");
+ ZebraStorageHint zebraStorageHint = ZebraStorageHint.createZebraStorageHint("[word]");
+ BasicTableOutputFormat.setStorageInfo(jobConf, zebraSchema, zebraStorageHint, null);
+
+ // set map-only job.
+ jobConf.setNumReduceTasks(0);
+ jobConf.setNumMapTasks(2);
+ JobClient.runJob(jobConf);
+ }
+ }
+}
+</source>
+ </section>
+ <!-- END ZEBRA OUTPUT EXAMPLE-->
+
+ <!-- ZEBRA SORT EXAMPLE-->
+<section>
+<title>Sort Columns</title>
+<p>This MapReduce code snippet demonstrates how to sort Zebra columns. </p>
+
+<source>
+/* user provides a Comparator Class for creating ZebraSortInfo */
+
+public static final class MemcmpRawComparator implements
+ RawComparator<Object>, Serializable {
+ @Override
+ public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
+ return WritableComparator.compareBytes(b1, s1, l1, b2, s2, l2);
+ }
+
+ @Override
+ public int compare(Object o1, Object o2) {
+ throw new RuntimeException("Object comparison not supported");
+ }
+}
+
+â¦
+â¦
+
+ZebraSchema zSchema = ZebraSchema.createZebraSchema(schemaString);
+
+ZebraStorageHint zStorageHint = ZebraStorageHint.createZebraStorageHint(storageHintString);
+
+/* Here we can use above Comparator Class to create a ZebraSortInfo object */
+
+ZebraSortInfo zSortInfo = ZebraSortInfo.createZebraSortInfo(sortColumnsString, MemcmpRawComparator.class);
+
+BasicTableOutputFormat.setStorageInfo(jobConf, zSchema, zStorageHint, zSortInfo);
+</source>
+</section>
+ <!-- END ZEBRA SORT EXAMPLE-->
+
+
+ <!-- ZEBRA DROP-CG EXAMPLE-->
+<section>
+<title>Drop Column Groups</title>
+<p>
+This example illustrates how to drop column groups (CG) in Zebra tables.
+This is not a MapReduce program since the API for deleting columg groups is a simple Java API.
+</p>
+
+<p>How to compile:</p>
+<source>
+# this command requires pig.jar and latest zebra jar.
+$ javac -cp pig.jar:zebra-0.1.0.jar DropColumnGroupExample.java
+
+# create a jar file
+$ jar cvf dropcg.jar DropColumnGroupExample.class
+</source>
+
+<p>How to run:</p>
+<source>
+# run the example.
+$ java -cp pig.jar:zebra-0.1.0.jar:dropcg.jar DropColumnGroupExample
+
+# This creates a table under the directory "dropCGExample".
+# If run the same command again, it fails since the destination
+# directory still exists. Please remove the directory before running.
+
+#This program takes one argument : directory for the example table
+$ java -cp pig.jar:zebra-0.1.0.jar:dropcg.jar DropColumnGroupExample tableDir
+</source>
+
+<p>Source code:</p>
+<source>
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.util.GenericOptionsParser;
+import org.apache.hadoop.zebra.io.BasicTable;
+import org.apache.hadoop.zebra.io.TableInserter;
+import org.apache.hadoop.zebra.io.TableScanner;
+import org.apache.hadoop.zebra.types.TypesUtils;
+import org.apache.pig.data.Tuple;
+
+public class DropColumnGroupExample {
+
+ private static String schema = "url, outcount:int, content:string";
+ private static String storageHint = "[url, outcount]; [content] as data";
+
+ public static void main(String[] args) throws Exception {
+
+ // Create configuration and process generaic Hadoop options like -D.
+ Configuration conf = new Configuration();
+ args = new GenericOptionsParser(conf, args).getRemainingArgs();
+
+ Path path = new Path((args.length > 0 ? args[0] : "dropCGExample"));
+
+ if (path.getFileSystem(conf).exists(path)) {
+ throw new IOException("Destination path '" + path + "' already exists. " +
+ "Please remove the directory and run again.");
+ }
+ // create a simple table.
+ createTable(conf, path);
+
+ //verify we can read the table.
+ String content = getContentForUrl(path, conf, "http://www.yahoo.com/");
+ if (content == null || !content.contains("yahoo")) {
+ throw new IOException("Table read failed.");
+ }
+
+ /* Now drop the colum group named "data".
+ *
+ * An exception is thrown if the CG could not be removed for some reason
+ * (e.g. the user might not have enough permissions).
+ */
+ BasicTable.dropColumnGroup(path, conf, "data");
+
+ // deleting an already deleted CG is a no-op.
+ BasicTable.dropColumnGroup(path, conf, "data");
+
+ // now try to read content column.
+ // Note that NULL is returned for the third column.
+ if (getContentForUrl(path, conf, "http://www.yahoo.com/") != null) {
+ throw new IOException("Expected NULL for 3rd column");
+ }
+ // While reading this table, a warning is logged since the user
+ // is trying to reading from a deleted CG.
+
+ //clean up the test directory.
+ //for now we are not deleting the directory to let users check it.
+ //BasicTable.drop(path, conf);
+ }
+
+ // Utility functions:
+
+ /**
+ * This is a simple table reader that iterates over the table to
+ * find a given url and returns its content.
+ */
+ private static String getContentForUrl(Path path, Configuration conf,
+ String url) throws Exception {
+
+ BasicTable.Reader reader = new BasicTable.Reader(path, conf);
+ TableScanner scanner = reader.getScanner(null, true);
+ Tuple row = TypesUtils.createTuple(3);
+
+ try {
+ while (!scanner.atEnd()) {
+ scanner.getValue(row);
+ if (url.equals(row.get(0))) {
+ return (String)row.get(2);
+ }
+ scanner.advance();
+ }
+ } finally {
+ scanner.close();
+ }
+ throw new IOException(url + " is not found");
+ }
+
+ private static void createTable(Configuration conf, Path path)
+ throws Exception {
+ /* NOTE: This creates a table using BasicTable API. This is not
+ * a committed public API yet. Typically tables are created
+ * in M/R or through PIG.
+ */
+ BasicTable.Writer writer = new BasicTable.Writer(path, schema,
+ storageHint, conf);
+ TableInserter inserter = writer.getInserter("part-0", true);
+ Tuple rowTuple = TypesUtils.createTuple(3);
+ BytesWritable emptyKey = new BytesWritable();
+
+ // add two rows:
+ rowTuple.set(0, "http://www.cnn.com/");
+ rowTuple.set(1, 10);
+ rowTuple.set(2, "content for cnn.com");
+
+ inserter.insert(emptyKey, rowTuple);
+
+ rowTuple.set(0, "http://www.yahoo.com/");
+ rowTuple.set(1, 20);
+ rowTuple.set(2, "content for yahoo.com");
+
+ inserter.insert(emptyKey, rowTuple);
+
+ inserter.close();
+ }
+}
+</source>
+</section>
+ <!-- END ZEBRA DROP-CG EXAMPLE-->
+
+ <!-- MULTIPLE TABLE OUTPUTS-->
+<section>
+<title>Multiple Table Outputs</title>
+<p>
+This code snippet illustrates how to work with multiple table outputs.
+</p>
+<source>
+In main()
+
+ String multiLocs = "/user/multi/us" + "," + "/user/multi/india" + "," + "/user/multi/japan";
+
+ jobConf.setOutputFormat(BasicTableOutputFormat.class);
+ BasicTableOutputFormat.setMultipleOutputPaths(jobConf, multiLocs);
+ BasicTableOutputFormat.setZebraOutputPartitionClass(jobConf, MultipleOutputsTest.OutputPartitionerClass.class);
+
+
+Implement a partition class:
+
+ static class OutputPartitionerClass implements ZebraOutputPartition {
+ @Override
+ public int getOutputPartition(BytesWritable key, Tuple value, String commaSeparatedLocs) {
+
+ String reg = null;
+ try {
+ reg = (String)(value.get(0));
+ } catch ( ExecException e) {
+ // do something about e
+ }
+
+ if(reg.equals("us")) return 0;
+ if(reg.equals("india")) return 1;
+ if(reg.equals("japan")) return 2;
+
+ return 0;
+ }
+ }
+</source>
+ </section>
+
+ <!-- endMULTIPLE TABLE OUTPUTS-->
+
+
+ </section>
+ <!--END ZEBRA M/R EXAMPLES-->
+
+ </body>
+ </document>
+
+
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_overview.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_overview.xml?rev=889870&view=auto
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_overview.xml (added)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_overview.xml Sat Dec 12 00:35:50 2009
@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!-- Copyright 2002-2004 The Apache Software Foundation
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+ "http://forrest.apache.org/dtd/document-v20.dtd">
+
+ <!-- BEGIN DOCUMENT-->
+
+<document>
+<header>
+<title>Overview</title>
+</header>
+<body>
+
+
+ <!-- OVERVIEW-->
+ <section>
+ <title>About Zebra</title>
+ <p>Zebra is an access path library for reading and writing data in a column-oriented fashion. Zebra functions as an abstraction layer between your client application and data on the Hadoop Distributed File System (HDFS). Data is written to HDFS using Zebraâs TableStore class. Data is read from HDFS using Zebraâs TableLoad class. Zebra supports client applications written as Pig, MapReduce, or streaming jobs. Keep in mind that Zebra works with Zebra tables; you cannot use Zebra to process text or sequence files.</p>
+<figure alt="Zebra" src="images/zebra.jpg" align="left"/>
+ </section>
+ <!-- END OVERVIEW-->
+
+
+ <!-- ZEBRA SETUP-->
+ <section>
+ <title>Zebra Setup</title>
+ <section>
+ <title>Prerequisites</title>
+ <p>Zebra requires:</p>
+ <ul>
+ <li>Pig 0.6.0 </li>
+ <li>Hadoop 0.20.1</li>
+ </ul>
+ <p></p>
+ <p>Also, make sure the following software is installed on your system:</p>
+ <ul>
+ <li>JDK 1.6</li>
+ <li>Ant 1.7.1</li>
+ <li>Javacc 4.2</li>
+ </ul>
+ </section>
+
+ <section>
+ <title>Download Zebra</title>
+<p>Zebra is a Pig contrib project and is available at: <br></br>
+ <a href="http://svn.apache.org/viewvc/hadoop/pig/trunk/contrib/zebra/">http://svn.apache.org/viewvc/hadoop/pig/trunk/contrib/zebra/</a></p>
+<p>To work with Zebra you need to check out the Pig trunk: <br></br>
+ <a href="http://svn.apache.org/repos/asf/hadoop/pig/trunk/ ">http://svn.apache.org/repos/asf/hadoop/pig/trunk/</a></p>
+ </section>
+
+ <section>
+ <title>Compile Zebra</title>
+ <p> To compile Zebra follow these steps.</p>
+
+<p>Step 1:</p>
+<ul>
+<li>Move to the top level of your Pig installation</li>
+<li>Run 'ant jar' (this builds the Pig classes and creates the Pig JAR files)</li>
+<li>(optional) Run 'ant -Dtestcase=none test-core' (this builds the Pig test classes which are needed by the Zebra tests)</li>
+</ul>
+ <p></p>
+<p>Step 2:</p>
+<ul>
+<li>cd ./contrib/zebra </li>
+<li>Run âant jarâ (this builds the Zebra classes and creates the Zebra JAR file)</li>
+<li>(optional) Run âant testâ (this verifies that Zebra is working correctly)</li>
+</ul>
+ </section>
+
+ </section>
+ <!-- END ZEBRA SETUP-->
+
+
+
+
+ </body>
+ </document>
+
+
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_pig.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_pig.xml?rev=889870&view=auto
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_pig.xml (added)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_pig.xml Sat Dec 12 00:35:50 2009
@@ -0,0 +1,189 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!-- Copyright 2002-2004 The Apache Software Foundation
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+ "http://forrest.apache.org/dtd/document-v20.dtd">
+
+ <!-- BEGIN DOCUMENT-->
+
+<document>
+<header>
+<title>Zebra and Pig</title>
+</header>
+<body>
+
+ <!-- OVERVIEW-->
+ <section>
+ <title>Overview</title>
+ <p>With Pig you can load and store data in Zebra format. You can also take advantage of sorted Zebra tables for map-side groups and merge joins. When working with Pig keep in mind that, unlike MapReduce, you do not need to declare Zebra schemas. Zebra automatically converts Zebra schemas to Pig schemas (and vice versa) for you.</p>
+
+ </section>
+ <!-- END OVERVIEW-->
+
+
+ <!-- ZEBRA EXAMPLES-->
+ <section>
+ <title>Zebra Pig Examples</title>
+<p></p>
+
+
+ <!--example: loading data-->
+ <section>
+ <title>Loading Data</title>
+ <p>To load data stored in Zebra format (columns) use the TableLoader function.</p>
+ <source>
+register $LOCATION/zebra-$version.jar;
+A = LOAD 'studenttab' USING org.apache.hadoop.zebra.pig.TableLoader();
+B = FOREACH A GENERATE name, age, gpa;
+</source>
+
+ <p>There are a couple of things to note:</p>
+ <ol>
+ <li>You need to register a Zebra jar file the same way you would do it for any other UDF.</li>
+ <li>You need to place the jar on your classpath.</li>
+ <li>When using Zebra with Pig, Zebra data is self-described and always contains a schema. This means that the AS clause is unnecessary as long as
+ you know what the column names and types are. To determine the column names and types, you can run the DESCRIBE statement right after the load:
+ <source>
+A = LOAD 'studenttab' USING org.apache.hadoop.zebra.pig.TableLoader();
+DESCRIBE A;
+a: {name: chararray,age: int,gpa: float}
+</source>
+ </li>
+ </ol>
+
+<p>You can provide alternative names to the columns with the AS clause. You can also provide types as long as the
+ original type can be converted to the new type. <em>In general</em>, Zebra supports Pig type compatibilities
+ (see <a href="piglatin_reference.html#Arithmetic+Operators+and+More">Arithmetic Operators and More</a>).</p>
+
+<p>You can provide multiple, comma-separated files to the loader:</p>
+<source>
+A = LOAD 'studenttab, votertab' USING org.apache.hadoop.zebra.pig.TableLoader();
+</source>
+
+<p>TableLoader supports efficient column selection; projections are automatically push down to the loader. This example tells the loader to only return two columns, name and age.</p>
+<source>
+A = LOAD 'studenttab' USING org.apache.hadoop.zebra.pig.TableLoader('name, age');
+</source>
+ </section>
+ <!--end example: loading data-->
+
+ <!--example: map-side data-->
+ <section>
+ <title>Map-Side Group and Merge Join</title>
+
+<p>If the input data is globally sorted, map-side group or merge join can be used. Please, notice the âsortedâ argument passed to the loader. This lets the loader know that the data is expected to be globally sorted and that a single key must be given to the same map.</p>
+
+<p>Here is an example of the merge join. Note that the first argument to the loader is left empty to indicate that all columns are requested.</p>
+<source>
+A = LOAD'studentsortedtab' USING org.apache.hadoop.zebra.pig.TableLoader('', 'sorted');
+B = LOAD 'votersortedtab' USING org.apache.hadoop.zebra.pig.TableLoader('', 'sorted');
+G = JOIN A BY $0, B By $0 USING "merge";
+</source>
+
+<p>Here is an example of a map-side group. Note that multiple sorted files are passed to the loader and that the loader will perform sort preserving merge to make sure that the data is globally sorted.</p>
+<source>
+A = LOAD 'studentsortedtab, studentnullsortedtab' using org.apache.hadoop.zebra.pig.TableLoader('name, age, gpa, source_table', 'sorted');
+B = GROUP A BY $0 USING "collected";
+C = FOREACH B GENERATE group, MAX(a.$1);
+</source>
+ </section>
+<!--end example: map-side data-->
+
+
+
+ <!--example: storing data-->
+ <section>
+ <title>Storing Data</title>
+ <p>To store data in Zebra format use the TableStorer function. Since Zebra does not support anonymous columns, the relation that is stored must have a name (alias) assigned to every column in the relation.</p>
+<source>
+A = LOAD 'studentsortedtab, studentnullsortedtab' USING org.apache.hadoop.zebra.pig.TableLoader('name, age, gpa, source_table', 'sorted');
+B = GROUP A BY $0 USING "collected";
+C = FOREACH B GENERATE group, MAX(a.$1) AS max_val;
+STORE C INTO 'output' USING org.apache.hadoop.zebra.pig.TableStorer('');
+</source>
+</section>
+ <!--end example: storing data-->
+
+
+
+ <!--example: simple types-->
+ <section>
+ <title>Simple Types</title>
+ <source>
+-- The following line is required if the Zebra jar is not part of Hadoop Core
+
+REGISTER $LOCATION/zebra-$version.jar;
+
+-- Load an existing table (a projection string is not specified so all the columns will be loaded)
+
+A = LOAD '$PATH/tbl1' USING org.apache.hadoop.pig.zebra.pig.TableLoader();
+
+-- Store in table format with storage hint '[c1];[c2]' which mean two Column Groups with one column each (note that you do not specify a schema)
+
+STORE A INTO '$PATH/tbl2' USING org.apache.hadoop.zebra.pig.TableStorer('[c1] COMPRESS BY gz; SERIALIZE BY pig;[c2]');
+
+-- Load an existing table (two columns are projected)
+
+B = LOAD '$PATH/tbl3' USING org.apache.hadoop.zebra.pig.TableLoader('c1,c2');
+
+-- Load an existing table (one column is projected)
+
+C = LOAD '$PATH/tbl4' USING org.apache.hadoop.zebra.pig.TableLoader('c2');
+ </source>
+ </section>
+ <!--end example: simple types-->
+
+
+ <!--example: complex types-->
+ <section>
+ <title>Complex Types</title>
+ <source>
+-- The following line is required if the Zebra jar is not part of Hadoop Core
+
+register $LOCATION/zebra-$version.jar;
+
+-- STR_SCHEMA =
+-- "s1:bool, s2:int, s3:long, s4:float, s5:string, s6:bytes,
+-- r1:record(f1:int, f2:long), r2:record(r3:record(f3:float, f4)),
+-- m1:map(string),m2:map(map(int)),
+-- c:collection(f13:double, f14:float, f15:bytes)";
+
+-- STR_STORAGE =
+-- "[s1, s2]; [m1#{a}]; [r1.f1]; [s3, s4, r2.r3.f3];
+-- [s5, s6, m2#{x|y}]; [r1.f2, m1#{b}]; [r2.r3.f4, m2#{z}]";
+
+
+-- Load an existing table (two columns are projected)
+
+A = LOAD '$PATH/tbl1' USING org.apache.hadoop.zebra.pig.TableLoader('r1.f2, s1');
+
+-- Store in table format (all columns are stored in one column group)
+
+STORE A INTO '$PATH/tbl2' USING org.apache.hadoop.zebra.pig.TableStorer('');
+
+-- Load an existing table (two columns are projected)
+
+B = LOAD '$PATH/tbl3' USING org.apache.hadoop.zebra.pig.TableLoader('s1, r1');
+ </source>
+ </section>
+ <!--end example: complex types-->
+ </section>
+ <!-- END ZEBRA EXAMPLES-->
+
+ </body>
+ </document>
+
+
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_reference.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_reference.xml?rev=889870&view=auto
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_reference.xml (added)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_reference.xml Sat Dec 12 00:35:50 2009
@@ -0,0 +1,700 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!-- Copyright 2002-2004 The Apache Software Foundation
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+ "http://forrest.apache.org/dtd/document-v20.dtd">
+
+ <!-- BEGIN DOCUMENT-->
+
+<document>
+<header>
+<title>Zebra Reference Guide</title>
+</header>
+<body>
+
+ <!-- ZEBRA TYPES-->
+ <section>
+ <title>Zebra Types</title>
+ <p>Zebra supports simple types (int, long, float, double, string, bytes), complex types (record, collection, map) and Booleans. </p>
+
+ <table>
+ <tr><th> Zebra Type</th><th> Description </th></tr>
+ <tr>
+ <td>int</td>
+ <td>signed 32-bit integer</td>
+ </tr>
+ <tr>
+ <td>long</td>
+ <td>signed 64-bit integer</td>
+ </tr>
+ <tr>
+ <td>float</td>
+ <td>32-bit floating point</td>
+ </tr>
+ <tr>
+ <td>double</td>
+ <td>64-bit floating point</td>
+ </tr>
+ <tr>
+ <td>string</td>
+ <td>character array (string) in Unicode UTF-8 format</td>
+ </tr>
+ <tr>
+ <td>bytes</td>
+ <td>byte array (blob)</td>
+ </tr>
+ <tr>
+ <td>record</td>
+ <td>An ordered set of fields. A field can be any Zebra type.</td>
+ </tr>
+ <tr>
+ <td>collection</td>
+ <td>A set of records. </td>
+ </tr>
+ <tr>
+ <td>map</td>
+ <td>A set of key/value pairs. The key is type string; the value can be any Zebra type.</td>
+ </tr>
+ <tr>
+ <td>bool</td>
+ <td>Boolean {0,1} false/true</td>
+ </tr>
+ </table>
+ <p></p>
+ <p>Zebra type names are chosen to be as âtechnology neutralâ as possible and are in resemblance to native types in modern programming languages.</p>
+ <table>
+ <tr><th> Zebra</th><th>Pig</th><th>Avro</th><th>SQL</th></tr>
+ <tr>
+ <td>int</td>
+ <td>int</td>
+ <td>int</td>
+ <td>integer</td>
+ </tr>
+ <tr>
+ <td>long</td>
+ <td>long</td>
+ <td>long</td>
+ <td>long</td>
+ </tr>
+ <tr>
+ <td>float</td>
+ <td>float</td>
+ <td>float</td>
+ <td>float,real</td>
+ </tr>
+ <tr>
+ <td>double</td>
+ <td>double</td>
+ <td>double</td>
+ <td>double precision</td>
+ </tr>
+ <tr>
+ <td>string</td>
+ <td>chararray</td>
+ <td>string</td>
+ <td>varchar</td>
+ </tr>
+ <tr>
+ <td>bytes</td>
+ <td>bytearray</td>
+ <td>bytes</td>
+ <td>raw</td>
+ </tr>
+ <tr>
+ <td>record</td>
+ <td>tuple</td>
+ <td>record</td>
+ <td>hash</td>
+ </tr>
+ <tr>
+ <td>collection</td>
+ <td>bag</td>
+ <td>array</td>
+ <td>list</td>
+ </tr>
+ <tr>
+ <td>map</td>
+ <td>map</td>
+ <td>map</td>
+ <td>hasp</td>
+ </tr>
+ <tr>
+ <td>bool</td>
+ <td>boolean</td>
+ <td>boolean</td>
+ <td>bool</td>
+ </tr>
+ </table>
+ </section>
+ <!-- END ZEBRA TYPES-->
+
+
+ <!-- STORE SCHEMA SPECIFICATION -->
+ <section>
+ <title>Store Schema Specification</title>
+<p>The Zebra store schema is used to write or store Zebra columns and to specify column types.
+The schema supports data type compatibility and conversion between Zebra/Pig, Zebra/MapReduce, and Zebra/Streaming.
+(<em>In a future release, the schema will also support type compatibility between Zebra/Pig-SQL and will guide the underlying serialization formats provided by Avro for projection, filtering, and so on. </em>)
+</p>
+
+ <p>The basic format for the the store schema is shown here.
+ The type name is optional; if not specified, the column defaults to type bytes.</p>
+ <p>
+<code>
+column_name[:type_name] [, column_name[:type_name] ... ]
+</code>
+</p>
+
+ <!-- SIMPLE TYPES -->
+ <section>
+ <title>Schemas for Simple Data Types</title>
+ <p>Simple data types include int, long, float, double, string, and bytes. The following syntax also applies to Booleans.</p>
+
+<section>
+ <title>Syntax</title>
+ <p>field_alias[:type] [, field_alias[:type] â¦]</p>
+</section>
+
+ <section>
+ <title>Terms</title>
+ <table>
+ <tr>
+ <td>field_alias</td>
+ <td>The name assigned to the field column.</td>
+ </tr>
+ <tr>
+ <td>:type</td>
+ <td>(Optional) The simple data type assigned to the field. <br></br>
+The alias and type are separated by a colon ( : ).<br></br>
+If the type is omitted, the field defaults to type bytes.
+</td>
+ </tr>
+ <tr>
+ <td> ,</td>
+ <td>Multiple fields are separated by commas.</td>
+ </tr>
+ </table>
+</section>
+
+ <section>
+ <title>Examples</title>
+ <p>In this example the schema specifies names and types for 3 columns.</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âs1:string, f1:float, i1:intâ);
+</source>
+ <p>In this example the schema specifies names for 3 columns; all 3 columns default to type bytes.</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âf1, f2, f3â);
+</source>
+</section>
+
+ </section>
+
+
+ <!-- RECORDS -->
+ <section>
+ <title>Schemas for Records</title>
+ <p>A record is an ordered set of fields. A field can be any Zebra type.</p>
+
+ <section>
+ <title>Syntax</title>
+ <p>record_alias:record (field_alias[:type]) [,(field_alias[:type]) â¦] )</p>
+</section>
+
+ <section>
+ <title>Terms</title>
+ <table>
+ <tr>
+ <td>record_alias</td>
+ <td>The name assigned to the record column.</td>
+ </tr>
+ <tr>
+ <td>:record</td>
+ <td>The record designator.</td>
+ </tr>
+ <tr>
+ <td>( )</td>
+ <td>The record notation, a set of parentheses.</td>
+ </tr>
+ <tr>
+ <td>field_alias</td>
+ <td>The name assigned to the field.</td>
+ </tr>
+ <tr>
+ <td>:type</td>
+ <td>(Optional) The type assigned to a field (can be any Zebra type).</td>
+ </tr>
+ <tr>
+ <td>,</td>
+ <td>Multiple fields are separated by commas.</td>
+ </tr>
+ </table>
+</section>
+
+ <section>
+ <title>Examples</title>
+ <p>In this example the schema specifies a record with two fields. </p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âr1:record(f1:int,f2:long)â);
+</source>
+ <p>In this example the schema specifies a record with two fields. Note that f2 will default to type bytes.</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âr1:record(r2:record(f1:int,f2)â);
+</source>
+</section>
+ </section>
+
+
+ <!-- COLLECTIONS -->
+ <section>
+ <title>Schemas for Collections</title>
+ <p>A collection is a set of records.</p>
+ <section>
+ <title>Syntax</title>
+ <p>collection_alias:collection (record)</p>
+</section>
+
+ <section>
+ <title>Terms</title>
+ <table>
+ <tr>
+ <td>collection_alias</td>
+ <td>The name assigned to the collection column.</td>
+ </tr>
+ <tr>
+ <td>:collection</td>
+ <td>The collection designator.</td>
+ </tr>
+ <tr>
+ <td>( )</td>
+ <td>The collection notation, a set of parentheses.</td>
+ </tr>
+ <tr>
+ <td> record</td>
+ <td>A record, specified one of two ways: <br></br>
+⢠Explicit (see Record)<br></br>
+c1:collection(r1:record(f1:int,f2:long))â);<br></br>
+⢠Implicit <br></br>
+c1:collection(f1:int,f2:long)â);
+</td>
+ </tr>
+ </table>
+</section>
+
+ <section>
+ <title>Examples</title>
+ <p>In this example the schema specifies a collection with a record consisting of two fields (explicit record notation).</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âc1:collection(r1:record(f1:int,f2:long))â);
+</source>
+ <p>In this example the schema specifies a collection with a record consisting of two fields (implicit record notation).</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âc1:collection(f1:int,f2:long)â);
+</source>
+</section>
+ </section>
+
+
+ <!-- MAPS -->
+ <section>
+ <title>Schemas for Maps</title>
+ <p>A map is a set of key/value pairs.</p>
+ <section>
+ <title>Syntax</title>
+ <p>map_alias:map (type)</p>
+</section>
+
+ <section>
+ <title>Terms</title>
+ <table>
+ <tr>
+ <td>map_alias</td>
+ <td>The name assigned to the map column.</td>
+ </tr>
+ <tr>
+ <td>:map</td>
+ <td>The map designator.</td>
+ </tr>
+ <tr>
+ <td>( )</td>
+ <td>The map notation, a set of parentheses.</td>
+ </tr>
+ <tr>
+ <td> type</td>
+ <td>The type assigned to the mapâs value (can be any Zebra type). <br></br>
+Note that the mapâs key is always type string and is not specified.
+</td>
+ </tr>
+ </table>
+</section>
+
+ <section>
+ <title>Examples</title>
+ <p>In this example the schema specifies a map with value of type string.</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âm1:map(string)â);
+</source>
+ <p>In this example the schema specifies a map with value of type map (with a value of type int).</p>
+<source>
+ZebraSchema.createZebraSchema(jobconf, âm2:map(map(int))â);
+</source>
+</section>
+ </section>
+ </section>
+ <!-- END STORE SCHEMA SPECIFICATION -->
+
+
+ <!-- STORAGE SPECIFICATION -->
+ <section>
+ <title>Storage Specification</title>
+
+ <p> The Zebra storage specification is used to specify column groups and the columns in those groups.
+</p>
+
+ <p>
+ The storage specification describes the physical structure of a Zebra table where $PATH/tbl indicates the table_directory and column_group_name
+ indicates the subdirectories within the table_directory. The STORE statement below indicates the following:
+ </p>
+ <ul>
+ <li>$PATH/mytable - directory</li>
+ <li>$PATH/mytable/A - subdirectory </li>
+ <li>$PATH/mytable/A/part00001 </li>
+ <li><em>etc ...</em></li>
+ </ul>
+
+<source>
+STORE A INTO '$PATH/mytable' USING org.apache.hadoop.zebra.pig.TableStorer('[a1, a2] AS A; [a3, a4, a5] AS B');
+</source>
+
+ <section>
+ <title>Specification</title>
+ <p> The basic format for the Zebra storage specification is shown here.
+ For this specification, note that the straight brackets <strong> [ ]</strong> designate a column group
+ and the curly brackets { } designate an optional syntax component.</p>
+
+<section>
+<title>Syntax</title>
+<p>
+<strong> [</strong>column_name {, column_name ...} <strong> ]</strong> {AS column_group_name} {COMPRESS BY compressor_name} {SERIALIZE BY serializer_name} <br></br>
+{; <strong> [</strong>column_name {, column_name ...} <strong> ]</strong> {AS column_group_name} {COMPRESS BY compressor_name} {SERIALIZE BY serializer_name} ... }
+</p>
+</section>
+
+<section>
+<title>Terms</title>
+ <table>
+ <tr>
+ <td><strong>[ ]</strong></td>
+ <td>Brackets designate a column group. Multiple column groups are separated by semi-colons.</td>
+ </tr>
+
+ <tr>
+ <td>AS column_group_name </td>
+ <td>Optional. The name of the column group. Column group names are unique within one table and are case sensitive: c1 and C1 are different. Column group names are used as the physical column group directory path names.
+ <br></br>If specified, the AS clause must immediately follow the column group [ ].
+ <br></br>If not specified, Zebra will assign unique default names for the table: CG0, CG1, CG2 ... (If CGx is specified by the programmer, then it cannot be used by Zebra.)
+ </td>
+ </tr>
+ <tr>
+ <td>COMPRESS BY compressor_name </td>
+ <td>Optional. Valid values for compressor_name include gz (default) and lzo. <br></br>If not specified, gz is used.</td>
+ </tr>
+
+ <tr>
+ <td>SERIALIZE BY serializer_name</td>
+ <td>Optional. Valid values for serializer_name include pig (default). (<em>In a future release, Avro will be available.</em>) <br></br>If not specified, pig is used.</td>
+ </tr>
+ <tr>
+ <td>column_name</td>
+ <td>The name of one or more columns that form the column group.</td>
+ </tr>
+ </table>
+
+</section>
+
+<section>
+<title>Examples</title>
+<p>In this example, one column group is specified; the two statements are equivalent.</p>
+<source>
+STORE A INTO '$PATH' USING org.apache.hadoop.zebra.pig.TableStorer('[c1]');
+
+STORE A INTO '$PATH' USING org.apache.hadoop.zebra.pig.TableStorer('[c1] AS CG0 COMPRESS BY gz SERIALIZE BY pig;');
+</source>
+
+<p>In this example, two column groups are specified. The first column group, C1, has two columns. The second column group, C2, has three columns.</p>
+<source>
+STORE A INTO '$PATH' USING org.apache.hadoop.zebra.pig.TableStorer('[a1, a2] AS C1; [a3, a4, a5] AS C2');
+</source>
+
+</section>
+</section>
+
+ </section>
+ <!-- END STORAGE SPECIFICATION -->
+
+
+ <!-- LOAD SCHEMA SPECIFICATION -->
+ <section>
+ <title>Load Schema Specification</title>
+
+ <p>The Zebra load schema is load or read table columns. </p>
+ <section>
+ <title>Schema</title>
+ <p>The basic format for the the Zebra load (read) schema is shown here. The column name can be any valid Zebra type.
+ If no columns are specified, the entire Zebra table is loaded.</p>
+ <p>
+<code>
+column_name [, column_name ... ]
+</code>
+ </p>
+
+<section>
+<title>Terms</title>
+ <table>
+ <tr>
+ <td>column_name</td>
+ <td>The column name. Multiple columns are separated by commas.</td>
+ </tr>
+ </table>
+</section>
+
+
+ <section>
+ <title>Example</title>
+
+ <p>Three Pig examples are shown here.</p>
+ <source>
+-- All columns are loaded
+A = LOAD '$PATH/tbl1' USING org.apache.hadoop.pig.zebra.pig.TableLoader();
+
+-- Two columns are projected
+B = LOAD '$PATH/tbl2' USING org.apache.hadoop.zebra.pig.TableLoader('c1,c2');
+
+-- Three columns are projected: a simple field, a map, and a record
+C = LOAD '$PATH/tbl3' USING org.apache.hadoop.zebra.pig.TableLoader('c1,c2#{key1},col4.{f1}')
+</source>
+ </section>
+ </section>
+ </section>
+ <!-- END LOAD SCHEMA SPECIFICATION -->
+
+ <!-- ZEBRA API-->
+ <section>
+ <title>Zebra MapReduce Interfaces</title>
+ <p>Zebra includes several classes for use in MapReduce programs. The main entry point into Zebra are the two classes for reading and writing tables, namely TableInputFormat and BasicTableOutputFormat. </p>
+
+ <section>
+ <title>BasicTableOutputFormat </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>yes</td>
+ <td>void setOutputPath(JobConf, Path) </td>
+ <td>Set the output path of the BasicTable in JobConf </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>Path[] getOutputPaths(JobConf) </td>
+ <td>Get the output paths of the BasicTable from JobConf </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>void setStorageInfo(JobConf, ZebraSchema, ZebraStorageHint, ZebraSortInfo) </td>
+ <td>Set the table storage information (schema, storagehint, sortinfo) in JobConf</td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>Schema getSchema(JobConf) </td>
+ <td>Get the table schema in JobConf </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>BytesWritable generateSortKey(JobConf, Tuple) </td>
+ <td>Generates a BytesWritable key for the input key </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>String getStorageHint(JobConf) </td>
+ <td>Get the table storage hint in JobConf </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>SortInfo getSortInfo(JobConf) </td>
+ <td>Get the SortInfo object </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>void close(JobConf) </td>
+ <td>Close the output BasicTable, No more rows can be added into the table </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>void setMultipleOutputs(JobConf, String commaSeparatedLocs, Class < extends ZebraOutputPartition> theClass) </td>
+ <td>Enables data to be written to multiple zebra tables based on the ZebraOutputPartition class.
+ See <a href="zebra_mapreduce.html#Multiple+Table+Outputs">Multiple Table Outputs.</a></td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>TableInputFormat </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>yes</td>
+ <td>void setInputPaths(JobConf, Path... paths) </td>
+ <td>Set the paths to the input table </td>
+
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>Path[] getInputPaths(JobConf) </td>
+ <td>Get the comma-separated paths to the input table or table union </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>Schema getSchema(JobConf) </td>
+ <td>Get the schema of a table expr </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>void setProjection(JobConf, ZebraProjection) </td>
+ <td>Set the input projection in the JobConf object </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>String getProjection(JobConf) </td>
+ <td>Get the projection from the JobConf </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>SortInfo getSortInfo(JobConf) </td>
+ <td>Get the SortInfo object regarding a Zebra table </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>void requireSortedTable(JobConf, String sortcolumns, BytesComparator comparator) </td>
+ <td>Requires sorted table or table union </td>
+ </tr>
+ <tr>
+ <td> yes </td>
+ <td>TableRecordReader getTableRecordReader(JobConf, ZebraProjection) </td>
+ <td>Get a TableRecordReader on a single split </td>
+ </tr>
+ <tr>
+ <td>yes</td>
+ <td>void setMinSplitSize(JobConf, long minSize) </td>
+ <td>Set the minimum split size, default of 1M bytes </td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>TableRecordReader </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>no</td>
+ <td>boolean seekTo(BytesWritable key) </td>
+ <td>Seek to the position at the first row which has the key (returning true) or just after the key(returning false); only applicable for sorted Zebra table. </td>
+ </tr>
+ </table>
+ </section>
+
+
+
+ <section>
+ <title>ZebraOutputPartition </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td> no </td>
+ <td>public abstract int getOutputPartition(BytesWritable key, Tuple value) </td>
+ <td>Abstract method from ZebraOutputPartition abstract class. App implements this to stream data to different table </td>
+ </tr>
+ <tr>
+ <td> no </td>
+ <td>void setConf(Configuration jobConf) </td>
+ <td>Initialization routine giving JobConf to application. Zebra implements it </td>
+ </tr>
+ <tr>
+ <td> no </td>
+ <td>Configuration getConf() </td>
+ <td> returns JobConf. Zebra implements it</td>
+ </tr>
+ <tr>
+ <td>yes </td>
+ <td>Class< extends ZebraOutputPartition> getZebraOutputPartitionClass(JobConf conf) </td>
+ <td>return user implemented ZebraOutputPartition class </td>
+ </tr>
+ </table>
+ </section>
+
+
+ <section>
+ <title>ZebraProjection </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>yes</td>
+ <td>ZebraProjection createZebraProjection(String) </td>
+ <td>Create a ZebraProjection object from a string representing projection information. </td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>ZebraSchema</title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>yes</td>
+ <td>ZebraSchema createZebraSchema(String) </td>
+ <td>Create a ZebraStorageHint object from a string representing storage hint information.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>ZebraStorageHint </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>yes</td>
+ <td>ZebraStorageHint createZebraStorageHint(String) </td>
+ <td>Create a ZebraStorageHint object from a string representing storage hint information. </td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>ZebraSortInfo </title>
+ <table>
+ <tr><th>Static</th><th>Method</th><th>Description</th></tr>
+ <tr>
+ <td>yes</td>
+ <td>ZebraSortInfo createZebraSortInfo(String sortColumns, Class< extends RawComparator < Object >> comparatorClass) </td>
+ <td>Create a ZebraSortInfo object from a sort columns string and a comparator class. </td>
+ </tr>
+ </table>
+ </section>
+ </section>
+ <!-- END ZEBRA API-->
+
+ </body>
+ </document>
+
+
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_stream.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_stream.xml?rev=889870&view=auto
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_stream.xml (added)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_stream.xml Sat Dec 12 00:35:50 2009
@@ -0,0 +1,189 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!-- Copyright 2002-2004 The Apache Software Foundation
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+ "http://forrest.apache.org/dtd/document-v20.dtd">
+
+ <!-- BEGIN DOCUMENT-->
+
+<document>
+<header>
+<title>Zebra and Streaming</title>
+</header>
+<body>
+
+
+ <!-- OVERVIEW-->
+ <section>
+ <title>Overview</title>
+ <p>Streaming allows you to write application logic in any langugage and to process large amounts
+ of data using the Hadoop framework. Streaming, which traditionally works with text files, can now be
+ used to process data stored as Zebra tables.</p>
+ </section>
+ <!-- END OVERVIEW-->
+
+ <!-- CONFIGURATION-->
+ <section>
+ <title>Configuration Variables</title>
+ <p>To use Zebra tables with your streaming applications, used the <code>mapred.lib.table.input.projection</code>
+ variable to specify Zebra columns (fields).</p>
+ <source>
+bin/hadoop jar $streamingJar -D mapred.lib.table.input.projection="word, count"
+</source>
+
+ </section>
+ <!-- END CONFIGURTION-->
+
+ <!-- ZEBRA EXAMPLES-->
+ <section>
+ <title>Zebra Streaming Examples</title>
+ <p>In the following examples, TableInputFormat is used for the
+ inputclass and the default TextOutputFormat is used for the outputclass. </p>
+
+ <!-- example: creating zebra table-->
+ <section>
+ <title>Creating a Zebra Table</title>
+ <p>Suppose a data file, testfile, contains four fields.</p>
+<source>
+en bbb1 1 1880
+en bbb2 1 2000
+</source>
+ <p>You can use a simple Pig script to create a Zebra table, testfile-table.
+ The table consists of one column group with four columns.</p>
+<source>
+$ cat table-creator.pig
+
+REGISTER $LOCATION/zebra-$version.jar;
+
+testfile = LOAD 'testfile'
+ USING PigStorage(' ') AS (language:chararray, page:chararray, count:int, size:long);
+
+STORE testfile INTO 'testfile-table'
+ USING org.apache.hadoop.zebra.pig.TableStorer('[language, page, count, size]');
+</source>
+ </section>
+ <!-- end example: creating zebra table-->
+
+
+ <!-- example: checking serialization-->
+ <section>
+ <title>Checking Serialization</title>
+ <p>This example is a map-only job that checks the serializtion. Note that each line starts
+ with a tab since the key is an empty string for tables created by PIG (this changes with sorted tables). </p>
+
+ <source>
+$ bin/hadoop jar hadoop-0.20.2-dev-streaming.jar -D mapred.reduce.tasks=0 \
+ -input testfile-table -output output -mapper 'cat' \
+ -inputformat org.apache.hadoop.zebra.mapred.TableInputFormat
+
+$ grep 'en' output/part-00000 | head
+
+(en,bbb1,1,1880)
+(en,bbb2,1,2000)
+(en,bbb3,1,1950)
+(en,bbb4,1,48900
+</source>
+ </section>
+ <!-- end example: checking serialization-->
+
+
+
+ <!-- example: locating frequently visited pages-->
+ <section>
+ <title>Locating Frequently Visited Pages</title>
+
+ <p>This perl script sorts the pages on number of page view counts. The script outputs space padded count
+ so that string sorting results in correct output. The first TAB separates the key and value for Hadoop streaming.</p>
+
+<source>
+while (<>) {
+
+ chomp;
+
+ s/.?\t(.*)$/$1/ or next; # ignore the key (if any) and remove braces
+
+ split ','; #comma seperated list.
+
+ # key is space padded 3rd column.
+
+ printf("%8d\t%s\n", $_[2], "@_") if @_ == 4; # without a projection
+
+ # printf("%8d\t%s\n", shift @_, join(',', @_)); # with projection="count, page"
+}
+</source>
+
+<p>Streaming command:</p>
+<source>
+$ bin/hadoop jar hadoop-0.20.2-dev-streaming.jar
+ -input testfile-table -output output -mapper table-mapper.pl -reducer cat \
+ -inputformat org.apache.hadoop.zebra.mapred.TableInputFormat
+</source>
+
+<p>Pages are printed in increasing order of page view counts.</p>
+<source>
+$ tail output/part-00000
+ 10 fr bbb1 10 5883
+ 14 de bbb2 14 2120
+ 20 it bbb3 20 229
+ 45 ja bbb4 45 75
+ 47 de bbb5 47 43488
+ 63 en bbb6 63 2404
+ 73 de bbb7 73 1090
+ 129 en bbb8 129 31
+ 188 en bbb9 188 37
+ 222 en bbb10 222 469
+</source>
+ </section>
+ <!-- end example: locating frequently visited pages-->
+
+
+ <!-- example: projecting columns-->
+ <section>
+ <title>Projecting Columns</title>
+
+ <p>Use projection to view only a few columns (fields) of a very large table. Modify the output line in
+ the table-mapper.pl script as shown below and run the following streaming command:</p>
+
+<source>
+$ bin/hadoop jar hadoop-0.20.2-dev-streaming.jar -D mapred.lib.table.input.projection="count,page" \
+ -input testfile-table -output output -mapper table-mapper.pl -reducer cat \
+ -inputformat org.apache.hadoop.zebra.mapred.TableInputFormat
+
+$ tail output/part-00000
+ 10 bbb1
+ 14 bbb2
+ 20 bbb3
+ 45 bbb4
+ 47 bbb5
+ 63 bbb6
+ 73 bbb7
+ 129 bbb8
+ 188 bbb9
+ 222 bbb10
+</source>
+ </section>
+ <!-- end example: projecting columns-->
+
+
+ </section>
+ <!-- END ZEBRA EXAMPLES-->
+
+
+
+ </body>
+ </document>
+
+
Added: hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_users.xml
URL: http://svn.apache.org/viewvc/hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_users.xml?rev=889870&view=auto
==============================================================================
--- hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_users.xml (added)
+++ hadoop/pig/trunk/src/docs/src/documentation/content/xdocs/zebra_users.xml Sat Dec 12 00:35:50 2009
@@ -0,0 +1,234 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!-- Copyright 2002-2004 The Apache Software Foundation
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+ "http://forrest.apache.org/dtd/document-v20.dtd">
+
+ <!-- BEGIN DOCUMENT-->
+
+<document>
+<header>
+<title>Zebra Users Guide</title>
+</header>
+<body>
+
+
+
+ <!-- COLUMN SECURITY-->
+ <section>
+ <title>Column Security</title>
+ <p><strong>NOTE: THIS FEATURE IS EXPERIMENTAL AND SUBJECT TO CHANGE IN THE FUTURE</strong></p>
+
+ <p>Since Zebra provides columnar storage of user data, we intend to separate secure and non-secure data into separate columns. We can then have access control based on HDFS file systesm based security. This would be achieved by an administrator setting appropriate permissions on the HDFS files contaning secure data. </p>
+
+ <section>
+ <title>Design Issues</title>
+<p>Roles:</p>
+<ul>
+<li>Publishers of secure data </li>
+<li>Consumers of secure data </li>
+<li>Administrators of secure data </li>
+</ul>
+
+ <p>How it will work:</p>
+ <ul>
+
+<li>Before any data is written in tables, all the corresponding files and directories need to have right set of ownership and permissions.
+This is necessary because if all the data is written and then the operation to make it secure is executed, that can leave security holes and paranoid will not allow that.</li>
+
+<li>All the files and direcories containing secure data will have same permissions and groups within a table </li>
+<li>User of the MR Job/Pig Script is required to have permissions to execute chgrp operations on a table. </li>
+<li>If no security information provided, then default behaviour.</li>
+<li>If permissions related error happens, it will be communicated to user as normal IOException </li>
+<li>If reader does not have read permissions for a column (CG), an IOException will be thrown </li>
+<li>The publisher/creator of the files will set these permissions when creating data.</li>
+</ul>
+ </section>
+
+<p></p>
+<p>One simple Pig example:</p>
+<source>
+a = load '/path_to_input_table' as (a:int, b:float,c:long,d:double);
+b = store a into '/path_to_output_table' using org.apache.hadoop.zebra.pig.TableStorer('[a, b] secure by group:secure perm:640');
+</source>
+
+<p>One simple MapReduce example:</p>
+<source>
+zStorageHint = ZebraStorageHint.createZebraStorageHint(â[a, b] secure by group:secure perm:640â);
+zSchema = â¦;
+zSortInfo = â¦;
+setStorageInfo(jobConf, zSchema, zStorageHint, zSortInf);
+</source>
+ </section>
+ <!-- END COLUMN SECURITY -->
+
+ <!-- DROP COLUMN GROUPS -->
+ <section>
+ <title>Drop Column Groups</title>
+ <p><strong>NOTE: THIS FEATURE IS EXPERIMENTAL AND SUBJECT TO CHANGE IN THE FUTURE</strong></p>
+
+ <p>Zebra allows you to delete a column group using the column group name.
+ For examples, see <a href="zebra_mapreduce.html#Drop+Column+Groups">Drop Column Groups</a>. </p>
+
+<p>Please note the following:</p>
+<ul>
+<li>Any failures during a drop will leave the table in consistent state (either with or with out the column group).
+While success of a CG removal guarantees a column removal, a failure does not imply CG is not removed.
+In rare cases, you might receive an error but the column could still be deleted. </li>
+<li>MapReduce jobs and other clients that are currently accessing the table might fail with exceptions.
+It is recommended that the column groups are dropped when there are no accesses to a table.
+It might not be feasible to ensure that there are no readers for a table; in these cases the readers should handle the exception. </li>
+<li>Once a column group is dropped, the column gruop data is deleted from the underlying filesystem.
+In the case of HDFS, it may not imply that physical data is actually removed because of earlier snapshot of the filesystem; handling this is out side the scope of Zebra. Legal requirements might require an admin finalize HDFS (if it is not already finalized) before or after performing a deletion. </li>
+<li>Concurrent deletions are supported and their access is serialized. </li>
+<li>Deleting a non-existant column group or a column group that is already deleted is not allowed.</li>
+<li>If you delete all the remaining columns in a table, it logically leaves an empty null table. The difference between a non-existant table and a table with zero columns is that opening a non-existant table causes an error. </li>
+ </ul>
+ </section>
+ <!-- END DROP COLUMN GROUPS -->
+
+ <!-- ORDER PRESERVE SORT-->
+ <section>
+ <title>Order-Preserving Sorted Table Union</title>
+<p>
+This Zebra functionality is only available on underlying sorted Zebra tables.
+</p>
+
+<section>
+<title>Output Records</title>
+<p>
+This feature groups all records from all "delta tables" on some sort key to form an output set of records while preserving the sorted ordering of the records in the origional tables. For instance, if the client application wants to fetch records from a union of tables of T1, T2 on a column "c1", then all records from T1 with a particular value of column "c1" and all records from T2 with that value of column "c1" will be output. The ordering of the rows of the output set of the same value of column "c1" is undefined. As a prerequisite, both T1 and T2 must be sorted on column "c1". More specifically the input and results could be as follows:
+</p>
+
+<p>Table T1: </p>
+<source>
+C1 C2
+--------------
+A 11
+A 12
+B 21
+B 22
+D 41
+</source>
+
+<p>Table T2: </p>
+<source>
+C1 C2
+-------------
+A 101
+A 102
+C 301
+D 401
+</source>
+
+<p>T1 Sort-Unioned with T2: </p>
+<source>
+source_table C1 C2
+---------------------------------------
+0 A 11
+1 A 101
+0 A 12
+1 A 102
+0 B 21
+0 B 22
+1 C 301
+1 D 401
+0 D 41
+</source>
+</section>
+
+<p>
+Note that the sortness is guaranteed per mapper and among all mappers arranged with certain ordering, but not among mappers arranged in any ordering. For instance, the outputs generated by 4 mappers, m1, m2, m3 and m4, could be in total ordering between m1, m3, m2 and m4, but not in any other arrangements.
+</p>
+
+<section>
+<title>Grouping and Indexing of Sort-Unioned Output Rows </title>
+<p>
+The order preserving sort-unioned results above could be further indexed by the component tables if the projection contains column(s) named "source_table". If so specified, the component table index will be output at the position(s) as specified in the projection list. If the underlying table is not a union of sorted tables, use of the special column name in projection will cause an exception thrown.
+</p>
+
+<p>
+If an attempt is made to create a table of a column named "source_table", an excpetion will be thrown as the name is reserved by zebra for the virtual name.
+</p>
+</section>
+
+<section>
+<title>MapReduce Interface </title>
+<p>
+TableInputFormat will have a static method, requireSortedTable, that allows the caller to specify the behavior of a single sorted table or an order preserving sorted table union as described above. The method will ensure all tables in a union are sorted. For more information, see <a href="zebra_reference.html#TableInputFormat">TableInputFormat</a>.
+</p>
+
+<p>One simple example: A order-preserving sorted union B. A and B are sorted tables. </p>
+<source>
+...
+TableInputFormat.setInputpaths("path_to_A, path_to_B");
+TableInputFormat.requireSortedTable();
+TableInputFormat.setProjection(conf, "f1, f2, source_table");
+...
+</source>
+</section>
+
+<section>
+<title>Pig Interface </title>
+<p>Pig will take an extra string argument of "sorted" indicating the desire to load from a sorted table or an order preserving sorted table union.
+For more information, see <a href="zebra_pig.html#Zebra+Pig+Examples">Zebra Pig Examples</a>.</p>
+
+<p>One simple example:</p>
+<source>
+...
+T = load ('path_to_A, path_to_B') using TableLoader('f1, f2, source_table', 'sorted');
+...
+</source>
+ </section>
+
+</section>
+<!-- END ORDER PRESERVE SORT-->
+
+ <!--MERGE JOIN-->
+ <section>
+ <title>Merge Join and Sorted Tables</title>
+
+<p>In data pipelines, there is often a need to join datasets. Zebra supports merge join on Zebra tables.
+For more information, see <a href="zebra_pig.html#Map-Side+Group+and+Merge+Join">Merge Join</a>.</p>
+
+<p>One simple example:</p>
+<source>
+Class myMapper {
+â¦
+Object keyGenerator;
+â¦
+
+public void map(â¦) {
+ bytesKey = BasicTableOutputFormat.getSortKey(keyGenerator, userKey);
+ â¦
+ output.collect(bytesKey, valueTuple);
+ â¦
+}
+
+public void configure(JobConf job)
+{
+ keyGenerator = BasicTableOutputFormat.getSortKeyGenerator(job);
+â¦
+}
+</source>
+
+ </section>
+ <!--END MERGE JOIN-->
+
+ </body>
+ </document>
+
+