You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by ch...@apache.org on 2014/05/17 02:08:58 UTC
svn commit: r1595392 - in /pig/branches/tez: ./ src/
src/docs/src/documentation/content/xdocs/ src/org/apache/pig/
src/org/apache/pig/backend/hadoop/executionengine/ src/org/apache/pig/impl/
src/org/apache/pig/newplan/logical/optimizer/ src/org/apache/...
Author: cheolsoo
Date: Sat May 17 00:08:58 2014
New Revision: 1595392
URL: http://svn.apache.org/r1595392
Log:
Merge latest trunk changes
Added:
pig/branches/tez/src/org/apache/pig/newplan/logical/visitor/DanglingNestedNodeRemover.java
- copied unchanged from r1595368, pig/trunk/src/org/apache/pig/newplan/logical/visitor/DanglingNestedNodeRemover.java
pig/branches/tez/src/org/apache/pig/newplan/logical/visitor/InputOutputFileValidatorVisitor.java
- copied unchanged from r1595368, pig/trunk/src/org/apache/pig/newplan/logical/visitor/InputOutputFileValidatorVisitor.java
Removed:
pig/branches/tez/src/org/apache/pig/newplan/logical/optimizer/DanglingNestedNodeRemover.java
pig/branches/tez/src/org/apache/pig/newplan/logical/rules/InputOutputFileValidator.java
Modified:
pig/branches/tez/ (props changed)
pig/branches/tez/CHANGES.txt
pig/branches/tez/src/docs/src/documentation/content/xdocs/perf.xml
pig/branches/tez/src/org/apache/pig/OverwritableStoreFunc.java
pig/branches/tez/src/org/apache/pig/PigServer.java
pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
pig/branches/tez/src/org/apache/pig/impl/PigContext.java
pig/branches/tez/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
pig/branches/tez/src/org/apache/pig/pen/ExampleGenerator.java
pig/branches/tez/src/pig-default.properties (props changed)
pig/branches/tez/test/org/apache/pig/parser/TestErrorHandling.java
pig/branches/tez/test/org/apache/pig/test/TestInputOutputFileValidator.java
pig/branches/tez/test/org/apache/pig/test/TestInputOutputMiniClusterFileValidator.java
pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java
pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java
pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java
pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushUpFilter.java
pig/branches/tez/test/org/apache/pig/test/TestPigStats.java
pig/branches/tez/test/org/apache/pig/test/TestProjectRange.java
pig/branches/tez/test/org/apache/pig/test/TestStore.java
pig/branches/tez/test/org/apache/pig/test/Util.java
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Arithmetic.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/BinCond.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Comparison.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/ComplexForeach.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Generate.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/IsNull1.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Split1.gld
pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Union.gld
pig/branches/tez/test/org/apache/pig/test/data/bzipdir1.bz2/bzipdir2.bz2/recordLossblockHeaderEndsAt136500.txt.bz2 (props changed)
Propchange: pig/branches/tez/
------------------------------------------------------------------------------
Merged /pig/trunk:r1593886-1595368
Modified: pig/branches/tez/CHANGES.txt
URL: http://svn.apache.org/viewvc/pig/branches/tez/CHANGES.txt?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/CHANGES.txt (original)
+++ pig/branches/tez/CHANGES.txt Sat May 17 00:08:58 2014
@@ -32,6 +32,10 @@ PIG-2207: Support custom counters for ag
IMPROVEMENTS
+PIG-3545: Seperate validation rules from optimizer (daijy)
+
+PIG-3745: Document auto local mode for pig (aniket486)
+
PIG-3932: Document ROUND_TO builtin UDF (mrflip via cheolsoo)
PIG-3926: ROUND_TO function: rounds double/float to fixed number of decimal places (mrflip via cheolsoo)
Modified: pig/branches/tez/src/docs/src/documentation/content/xdocs/perf.xml
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/docs/src/documentation/content/xdocs/perf.xml?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/docs/src/documentation/content/xdocs/perf.xml (original)
+++ pig/branches/tez/src/docs/src/documentation/content/xdocs/perf.xml Sat May 17 00:08:58 2014
@@ -1075,6 +1075,24 @@ Direct fetch is turned on by default. To
</p>
</section>
+
+<!-- +++++++++++++++++++++++++++++++ -->
+<section id="auto-local-mode">
+<title>Auto Local Mode</title>
+<p>Processing small mapreduce jobs on hadoop cluster could be slow as it has overhead of job startup and job scheduling. For jobs with small input data, pig can convert them to run them as in-process mapreduce with hadoop's local mode. If pig.auto.local.enabled flag is set to true, pig will convert mapreduce jobs with input data less than pig.auto.local.input.maxbytes (100MB by default) to run in local mode, provided the number of reducers required by the job are less than or equal to 1. Note, jobs converted to run in local mode load and store data from HDFS, so any job in the pig workflow(dag) could be converted to run in local mode without affecting its downstream jobs.</p>
+
+<p>You can set the values for these properties in order to configure the behavior:</p>
+
+<ul>
+<li>pig.auto.local.enabled - Turns on/off auto local mode feature (false by default).</li>
+<li>pig.auto.local.input.maxbytes - Controls the max threshold size (in bytes) to convert jobs to run in local mode (100MB by default).</li>
+</ul>
+<p></p>
+<p>
+Sometimes, you may want change job configuration for jobs that are converted to run in local mode (eg- change io.sort.mb for small jobs). To do so, you can use pig.local. prefix to any configuration and configuration will be set on converted jobs. For example, set pig.local.io.sort.mb 100 will change io.sort.mb value to 100 for jobs converted to run in local mode.
+</p>
+
+</section>
</section>
<!-- ==================================================================== -->
Modified: pig/branches/tez/src/org/apache/pig/OverwritableStoreFunc.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/org/apache/pig/OverwritableStoreFunc.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/org/apache/pig/OverwritableStoreFunc.java (original)
+++ pig/branches/tez/src/org/apache/pig/OverwritableStoreFunc.java Sat May 17 00:08:58 2014
@@ -22,7 +22,6 @@ import java.io.IOException;
import org.apache.hadoop.mapreduce.Job;
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat;
import org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore;
-import org.apache.pig.newplan.logical.rules.InputOutputFileValidator;
/**
* A {@link StoreFunc} should implement this interface to enable overwriting its
Modified: pig/branches/tez/src/org/apache/pig/PigServer.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/org/apache/pig/PigServer.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/org/apache/pig/PigServer.java (original)
+++ pig/branches/tez/src/org/apache/pig/PigServer.java Sat May 17 00:08:58 2014
@@ -63,8 +63,6 @@ import org.apache.pig.impl.io.FileLocali
import org.apache.pig.impl.io.FileLocalizer.FetchFileRet;
import org.apache.pig.impl.logicalLayer.FrontendException;
import org.apache.pig.impl.logicalLayer.schema.Schema;
-import org.apache.pig.impl.plan.CompilationMessageCollector;
-import org.apache.pig.impl.plan.CompilationMessageCollector.MessageType;
import org.apache.pig.impl.streaming.StreamingCommand;
import org.apache.pig.impl.util.LogUtils;
import org.apache.pig.impl.util.PropertiesUtil;
@@ -78,7 +76,6 @@ import org.apache.pig.newplan.logical.ex
import org.apache.pig.newplan.logical.expression.LogicalExpressionVisitor;
import org.apache.pig.newplan.logical.expression.ScalarExpression;
import org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor;
-import org.apache.pig.newplan.logical.optimizer.DanglingNestedNodeRemover;
import org.apache.pig.newplan.logical.relational.LOForEach;
import org.apache.pig.newplan.logical.relational.LOLoad;
import org.apache.pig.newplan.logical.relational.LOStore;
@@ -86,15 +83,6 @@ import org.apache.pig.newplan.logical.re
import org.apache.pig.newplan.logical.relational.LogicalPlanData;
import org.apache.pig.newplan.logical.relational.LogicalRelationalOperator;
import org.apache.pig.newplan.logical.relational.LogicalSchema;
-import org.apache.pig.newplan.logical.visitor.CastLineageSetter;
-import org.apache.pig.newplan.logical.visitor.ColumnAliasConversionVisitor;
-import org.apache.pig.newplan.logical.visitor.DuplicateForEachColumnRewriteVisitor;
-import org.apache.pig.newplan.logical.visitor.ImplicitSplitInsertVisitor;
-import org.apache.pig.newplan.logical.visitor.ScalarVariableValidator;
-import org.apache.pig.newplan.logical.visitor.ScalarVisitor;
-import org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor;
-import org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor;
-import org.apache.pig.newplan.logical.visitor.UnionOnSchemaSetter;
import org.apache.pig.parser.QueryParserDriver;
import org.apache.pig.parser.QueryParserUtils;
import org.apache.pig.pen.ExampleGenerator;
@@ -107,7 +95,6 @@ import org.apache.pig.tools.pigstats.Pig
import org.apache.pig.tools.pigstats.PigStats.JobGraph;
import org.apache.pig.tools.pigstats.ScriptState;
import org.apache.pig.validator.BlackAndWhitelistFilter;
-import org.apache.pig.validator.BlackAndWhitelistValidator;
import org.apache.pig.validator.PigCommandFilter;
import com.google.common.annotations.VisibleForTesting;
@@ -155,8 +142,6 @@ public class PigServer {
protected final String scope = constructScope();
- private boolean aggregateWarning = true;
-
private boolean validateEachStatement = false;
private boolean skipParseInRegisterForBatch = false;
@@ -233,8 +218,6 @@ public class PigServer {
this.pigContext = context;
currDAG = new Graph(false);
- aggregateWarning = "true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
-
jobName = pigContext.getProperties().getProperty(
PigContext.JOB_NAME,
PigContext.JOB_NAME_PREFIX + ":DefaultJobName");
@@ -804,6 +787,7 @@ public class PigServer {
*/
public Schema dumpSchema(String alias) throws IOException {
try {
+ pigContext.inDumpSchema = true;
if ("@".equals(alias)) {
alias = getLastRel();
}
@@ -825,6 +809,8 @@ public class PigServer {
int errCode = 1001;
String msg = "Unable to describe schema for alias " + alias;
throw new FrontendException (msg, errCode, PigException.INPUT, false, null, fee);
+ } finally {
+ pigContext.inDumpSchema = false;
}
}
@@ -838,26 +824,31 @@ public class PigServer {
* @throws IOException
*/
public Schema dumpSchemaNested(String alias, String nestedAlias) throws IOException {
- if ("@".equals(alias)) {
- alias = getLastRel();
- }
- Operator op = getOperatorForAlias( alias );
- if( op instanceof LOForEach ) {
- LogicalSchema nestedSc = ((LOForEach)op).dumpNestedSchema(alias, nestedAlias);
- if (nestedSc!=null) {
- Schema s = org.apache.pig.newplan.logical.Util.translateSchema(nestedSc);
- System.out.println(alias+ "::" + nestedAlias + ": " + s.toString());
- return s;
+ try {
+ pigContext.inDumpSchema = true;
+ if ("@".equals(alias)) {
+ alias = getLastRel();
+ }
+ Operator op = getOperatorForAlias( alias );
+ if( op instanceof LOForEach ) {
+ LogicalSchema nestedSc = ((LOForEach)op).dumpNestedSchema(alias, nestedAlias);
+ if (nestedSc!=null) {
+ Schema s = org.apache.pig.newplan.logical.Util.translateSchema(nestedSc);
+ System.out.println(alias+ "::" + nestedAlias + ": " + s.toString());
+ return s;
+ }
+ else {
+ System.out.println("Schema for "+ alias+ "::" + nestedAlias + " unknown.");
+ return null;
+ }
}
else {
- System.out.println("Schema for "+ alias+ "::" + nestedAlias + " unknown.");
- return null;
+ int errCode = 1001;
+ String msg = "Unable to describe schema for " + alias + "::" + nestedAlias;
+ throw new FrontendException (msg, errCode, PigException.INPUT, false, null);
}
- }
- else {
- int errCode = 1001;
- String msg = "Unable to describe schema for " + alias + "::" + nestedAlias;
- throw new FrontendException (msg, errCode, PigException.INPUT, false, null);
+ } finally {
+ pigContext.inDumpSchema = false;
}
}
@@ -1076,6 +1067,7 @@ public class PigServer {
try {
pigContext.inExplain = true;
buildStorePlan( alias );
+ currDAG.lp.optimize(pigContext);
//Only add root xml node if all plans are being written to same stream.
if (format == "xml" && lps == eps) {
@@ -1366,9 +1358,7 @@ public class PigServer {
FrontendException {
// discover pig features used in this script
ScriptState.get().setScriptFeatures(currDAG.lp);
-
- BlackAndWhitelistValidator validator = new BlackAndWhitelistValidator(getPigContext(), currDAG.lp);
- validator.validate();
+ currDAG.lp.optimize(pigContext);
return launchPlan(currDAG.lp, "job_pigexec_");
}
@@ -1703,7 +1693,7 @@ public class PigServer {
QueryParserDriver parserDriver = new QueryParserDriver( pigContext, scope, fileNameMap );
try {
LogicalPlan plan = parserDriver.parse( query );
- compile( plan );
+ plan.validate(pigContext, scope);
} catch(FrontendException ex) {
scriptCache.remove( scriptCache.size() -1 );
throw ex;
@@ -1762,43 +1752,10 @@ public class PigServer {
}
private void compile() throws IOException {
- compile( lp );
+ lp.validate(pigContext, scope);
currDAG.postProcess();
}
- private void compile(LogicalPlan lp) throws FrontendException {
- DanglingNestedNodeRemover DanglingNestedNodeRemover = new DanglingNestedNodeRemover( lp );
- DanglingNestedNodeRemover.visit();
-
- new ColumnAliasConversionVisitor(lp).visit();
- new SchemaAliasVisitor(lp).visit();
- new ScalarVisitor(lp, pigContext, scope).visit();
-
- // ImplicitSplitInsertVisitor has to be called before
- // DuplicateForEachColumnRewriteVisitor. Detail at pig-1766
- new ImplicitSplitInsertVisitor(lp).visit();
-
- // DuplicateForEachColumnRewriteVisitor should be before
- // TypeCheckingRelVisitor which does resetSchema/getSchema
- // heavily
- new DuplicateForEachColumnRewriteVisitor(lp).visit();
-
- CompilationMessageCollector collector = new CompilationMessageCollector() ;
-
- new TypeCheckingRelVisitor( lp, collector).visit();
- new UnionOnSchemaSetter( lp ).visit();
- new CastLineageSetter(lp, collector).visit();
- new ScalarVariableValidator(lp).visit();
- if(aggregateWarning) {
- CompilationMessageCollector.logMessages(collector, MessageType.Warning, aggregateWarning, log);
- } else {
- for(Enum type: MessageType.values()) {
- CompilationMessageCollector.logAllMessages(collector, log);
- }
- }
-
- }
-
private void postProcess() throws IOException {
// The following code deals with store/load combination of
// intermediate files. In this case we will replace the load
Modified: pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java (original)
+++ pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java Sat May 17 00:08:58 2014
@@ -22,7 +22,6 @@ import java.io.File;
import java.io.IOException;
import java.io.PrintStream;
import java.net.URL;
-import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Properties;
@@ -32,7 +31,6 @@ import org.apache.commons.logging.LogFac
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.mapred.JobConf;
-import org.apache.pig.PigConstants;
import org.apache.pig.PigException;
import org.apache.pig.backend.BackendException;
import org.apache.pig.backend.datastorage.DataStorage;
@@ -47,32 +45,19 @@ import org.apache.pig.backend.hadoop.exe
import org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil;
import org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager;
import org.apache.pig.impl.PigContext;
-import org.apache.pig.impl.PigImplConstants;
import org.apache.pig.impl.logicalLayer.FrontendException;
import org.apache.pig.impl.plan.OperatorKey;
import org.apache.pig.impl.plan.PlanException;
import org.apache.pig.impl.plan.VisitorException;
import org.apache.pig.impl.streaming.ExecutableManager;
-import org.apache.pig.impl.util.ObjectSerializer;
import org.apache.pig.impl.util.Utils;
import org.apache.pig.newplan.Operator;
-import org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer;
-import org.apache.pig.newplan.logical.optimizer.SchemaResetter;
-import org.apache.pig.newplan.logical.optimizer.UidResetter;
import org.apache.pig.newplan.logical.relational.LOForEach;
import org.apache.pig.newplan.logical.relational.LogToPhyTranslationVisitor;
import org.apache.pig.newplan.logical.relational.LogicalPlan;
import org.apache.pig.newplan.logical.relational.LogicalRelationalOperator;
-import org.apache.pig.newplan.logical.rules.InputOutputFileValidator;
-import org.apache.pig.newplan.logical.rules.LogicalRelationalNodeValidator;
-import org.apache.pig.newplan.logical.visitor.SortInfoSetter;
-import org.apache.pig.newplan.logical.visitor.StoreAliasSetter;
-import org.apache.pig.pen.POOptimizeDisabler;
import org.apache.pig.tools.pigstats.PigStats;
-import org.apache.pig.validator.BlackAndWhitelistValidator;
-import com.google.common.base.Splitter;
-import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
public abstract class HExecutionEngine implements ExecutionEngine {
@@ -100,7 +85,6 @@ public abstract class HExecutionEngine i
// val: the operator key for the root of the phyisical plan
protected Map<OperatorKey, OperatorKey> logicalToPhysicalKeys;
protected Map<Operator, PhysicalOperator> newLogToPhyMap;
- private LogicalPlan newPreoptimizedPlan;
public HExecutionEngine(PigContext pigContext) {
this.pigContext = pigContext;
@@ -245,82 +229,6 @@ public abstract class HExecutionEngine i
throw new FrontendException(msg, errCode, PigException.BUG);
}
- newPreoptimizedPlan = new LogicalPlan(plan);
-
- if (pigContext.inIllustrator) {
- // disable all PO-specific optimizations
- POOptimizeDisabler pod = new POOptimizeDisabler(plan);
- pod.visit();
- }
-
- UidResetter uidResetter = new UidResetter(plan);
- uidResetter.visit();
-
- SchemaResetter schemaResetter = new SchemaResetter(plan,
- true /* skip duplicate uid check*/);
- schemaResetter.visit();
-
- HashSet<String> disabledOptimizerRules;
- try {
- disabledOptimizerRules = (HashSet<String>) ObjectSerializer
- .deserialize(pigContext.getProperties().getProperty(
- PigImplConstants.PIG_OPTIMIZER_RULES_KEY));
- } catch (IOException ioe) {
- int errCode = 2110;
- String msg = "Unable to deserialize optimizer rules.";
- throw new FrontendException(msg, errCode, PigException.BUG, ioe);
- }
- if (disabledOptimizerRules == null) {
- disabledOptimizerRules = new HashSet<String>();
- }
-
- String pigOptimizerRulesDisabled = this.pigContext.getProperties()
- .getProperty(PigConstants.PIG_OPTIMIZER_RULES_DISABLED_KEY);
- if (pigOptimizerRulesDisabled != null) {
- disabledOptimizerRules.addAll(Lists.newArrayList((Splitter.on(",")
- .split(pigOptimizerRulesDisabled))));
- }
-
- if (pigContext.inIllustrator) {
- disabledOptimizerRules.add("MergeForEach");
- disabledOptimizerRules.add("PartitionFilterOptimizer");
- disabledOptimizerRules.add("LimitOptimizer");
- disabledOptimizerRules.add("SplitFilter");
- disabledOptimizerRules.add("PushUpFilter");
- disabledOptimizerRules.add("MergeFilter");
- disabledOptimizerRules.add("PushDownForEachFlatten");
- disabledOptimizerRules.add("ColumnMapKeyPrune");
- disabledOptimizerRules.add("AddForEach");
- disabledOptimizerRules.add("GroupByConstParallelSetter");
- }
-
- StoreAliasSetter storeAliasSetter = new StoreAliasSetter(plan);
- storeAliasSetter.visit();
-
- // run optimizer
- LogicalPlanOptimizer optimizer = new LogicalPlanOptimizer(plan, 100,
- disabledOptimizerRules);
- optimizer.optimize();
-
- // compute whether output data is sorted or not
- SortInfoSetter sortInfoSetter = new SortInfoSetter(plan);
- sortInfoSetter.visit();
-
- if (!pigContext.inExplain) {
- // Validate input/output file. Currently no validation framework in
- // new logical plan, put this validator here first.
- // We might decide to move it out to a validator framework in future
- LogicalRelationalNodeValidator validator = new InputOutputFileValidator(
- plan, pigContext);
- validator.validate();
-
- // Check for blacklist and whitelist properties and disable
- // commands/operators accordingly. Note if a user does not
- // specify these, Pig will work without any filters or validations
- validator = new BlackAndWhitelistValidator(pigContext, plan);
- validator.validate();
- }
-
// translate new logical plan to physical plan
LogToPhyTranslationVisitor translator = new LogToPhyTranslationVisitor(plan);
@@ -354,11 +262,7 @@ public abstract class HExecutionEngine i
}
return result;
}
-
- public LogicalPlan getNewPlan() {
- return newPreoptimizedPlan;
- }
-
+
public PigStats launchPig(LogicalPlan lp, String grpName, PigContext pc)
throws FrontendException, ExecException {
Modified: pig/branches/tez/src/org/apache/pig/impl/PigContext.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/org/apache/pig/impl/PigContext.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/org/apache/pig/impl/PigContext.java (original)
+++ pig/branches/tez/src/org/apache/pig/impl/PigContext.java Sat May 17 00:08:58 2014
@@ -162,6 +162,9 @@ public class PigContext implements Seria
// existence checks, etc).
public boolean inExplain = false;
+ // Where we are processing a dump schema right now
+ public boolean inDumpSchema = false;
+
// whether we're processing an ILLUSTRATE right now.
public boolean inIllustrator = false;
Modified: pig/branches/tez/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java (original)
+++ pig/branches/tez/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java Sat May 17 00:08:58 2014
@@ -18,19 +18,48 @@
package org.apache.pig.newplan.logical.relational;
+import java.io.IOException;
import java.io.PrintStream;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
+import org.apache.pig.PigConstants;
+import org.apache.pig.PigException;
+import org.apache.pig.impl.PigContext;
+import org.apache.pig.impl.PigImplConstants;
import org.apache.pig.impl.logicalLayer.FrontendException;
+import org.apache.pig.impl.plan.CompilationMessageCollector;
+import org.apache.pig.impl.plan.CompilationMessageCollector.MessageType;
import org.apache.pig.impl.util.HashOutputStream;
+import org.apache.pig.impl.util.ObjectSerializer;
import org.apache.pig.newplan.BaseOperatorPlan;
import org.apache.pig.newplan.Operator;
import org.apache.pig.newplan.OperatorPlan;
import org.apache.pig.newplan.logical.DotLOPrinter;
+import org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer;
import org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter;
+import org.apache.pig.newplan.logical.optimizer.SchemaResetter;
+import org.apache.pig.newplan.logical.optimizer.UidResetter;
+import org.apache.pig.newplan.logical.visitor.CastLineageSetter;
+import org.apache.pig.newplan.logical.visitor.ColumnAliasConversionVisitor;
+import org.apache.pig.newplan.logical.visitor.DanglingNestedNodeRemover;
+import org.apache.pig.newplan.logical.visitor.DuplicateForEachColumnRewriteVisitor;
+import org.apache.pig.newplan.logical.visitor.ImplicitSplitInsertVisitor;
+import org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor;
+import org.apache.pig.newplan.logical.visitor.ScalarVariableValidator;
+import org.apache.pig.newplan.logical.visitor.ScalarVisitor;
+import org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor;
+import org.apache.pig.newplan.logical.visitor.SortInfoSetter;
+import org.apache.pig.newplan.logical.visitor.StoreAliasSetter;
+import org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor;
+import org.apache.pig.newplan.logical.visitor.UnionOnSchemaSetter;
+import org.apache.pig.pen.POOptimizeDisabler;
+import org.apache.pig.validator.BlackAndWhitelistValidator;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Lists;
import com.google.common.hash.HashFunction;
import com.google.common.hash.Hashing;
@@ -136,4 +165,106 @@ public class LogicalPlan extends BaseOpe
return Integer.toString(hos.getHashCode().asInt());
}
+
+ public void validate(PigContext pigContext, String scope) throws FrontendException {
+
+ new DanglingNestedNodeRemover(this).visit();
+ new ColumnAliasConversionVisitor(this).visit();
+ new SchemaAliasVisitor(this).visit();
+ new ScalarVisitor(this, pigContext, scope).visit();
+
+ // ImplicitSplitInsertVisitor has to be called before
+ // DuplicateForEachColumnRewriteVisitor. Detail at pig-1766
+ new ImplicitSplitInsertVisitor(this).visit();
+
+ // DuplicateForEachColumnRewriteVisitor should be before
+ // TypeCheckingRelVisitor which does resetSchema/getSchema
+ // heavily
+ new DuplicateForEachColumnRewriteVisitor(this).visit();
+
+ CompilationMessageCollector collector = new CompilationMessageCollector() ;
+
+ new TypeCheckingRelVisitor( this, collector).visit();
+
+ boolean aggregateWarning = "true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
+
+ if(aggregateWarning) {
+ CompilationMessageCollector.logMessages(collector, MessageType.Warning, aggregateWarning, log);
+ } else {
+ for(Enum type: MessageType.values()) {
+ CompilationMessageCollector.logAllMessages(collector, log);
+ }
+ }
+
+ new UnionOnSchemaSetter(this).visit();
+ new CastLineageSetter(this, collector).visit();
+ new ScalarVariableValidator(this).visit();
+ new StoreAliasSetter(this).visit();
+
+ // compute whether output data is sorted or not
+ new SortInfoSetter(this).visit();
+
+ if (!(pigContext.inExplain || pigContext.inDumpSchema)) {
+ // Validate input/output file
+ new InputOutputFileValidatorVisitor(this, pigContext).visit();
+ }
+
+ BlackAndWhitelistValidator validator = new BlackAndWhitelistValidator(pigContext, this);
+ validator.validate();
+
+ // Now make sure the plan is consistent
+ UidResetter uidResetter = new UidResetter(this);
+ uidResetter.visit();
+
+ SchemaResetter schemaResetter = new SchemaResetter(this,
+ true /* skip duplicate uid check*/);
+ schemaResetter.visit();
+ }
+
+ public void optimize(PigContext pigContext) throws FrontendException {
+ if (pigContext.inIllustrator) {
+ // disable all PO-specific optimizations
+ POOptimizeDisabler pod = new POOptimizeDisabler(this);
+ pod.visit();
+ }
+
+ HashSet<String> disabledOptimizerRules;
+ try {
+ disabledOptimizerRules = (HashSet<String>) ObjectSerializer
+ .deserialize(pigContext.getProperties().getProperty(
+ PigImplConstants.PIG_OPTIMIZER_RULES_KEY));
+ } catch (IOException ioe) {
+ int errCode = 2110;
+ String msg = "Unable to deserialize optimizer rules.";
+ throw new FrontendException(msg, errCode, PigException.BUG, ioe);
+ }
+ if (disabledOptimizerRules == null) {
+ disabledOptimizerRules = new HashSet<String>();
+ }
+
+ String pigOptimizerRulesDisabled = pigContext.getProperties()
+ .getProperty(PigConstants.PIG_OPTIMIZER_RULES_DISABLED_KEY);
+ if (pigOptimizerRulesDisabled != null) {
+ disabledOptimizerRules.addAll(Lists.newArrayList((Splitter.on(",")
+ .split(pigOptimizerRulesDisabled))));
+ }
+
+ if (pigContext.inIllustrator) {
+ disabledOptimizerRules.add("MergeForEach");
+ disabledOptimizerRules.add("PartitionFilterOptimizer");
+ disabledOptimizerRules.add("LimitOptimizer");
+ disabledOptimizerRules.add("SplitFilter");
+ disabledOptimizerRules.add("PushUpFilter");
+ disabledOptimizerRules.add("MergeFilter");
+ disabledOptimizerRules.add("PushDownForEachFlatten");
+ disabledOptimizerRules.add("ColumnMapKeyPrune");
+ disabledOptimizerRules.add("AddForEach");
+ disabledOptimizerRules.add("GroupByConstParallelSetter");
+ }
+
+ // run optimizer
+ LogicalPlanOptimizer optimizer = new LogicalPlanOptimizer(this, 100,
+ disabledOptimizerRules);
+ optimizer.optimize();
+ }
}
Modified: pig/branches/tez/src/org/apache/pig/pen/ExampleGenerator.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/src/org/apache/pig/pen/ExampleGenerator.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/src/org/apache/pig/pen/ExampleGenerator.java (original)
+++ pig/branches/tez/src/org/apache/pig/pen/ExampleGenerator.java Sat May 17 00:08:58 2014
@@ -230,8 +230,9 @@ public class ExampleGenerator {
}
PhysicalPlan compilePlan(LogicalPlan plan) throws ExecException, FrontendException {
+ newPlan = new LogicalPlan(plan);
+ plan.optimize(pigContext);
PhysicalPlan result = execEngine.compile(plan, null);
- newPlan = execEngine.getNewPlan();
return result;
}
Propchange: pig/branches/tez/src/pig-default.properties
------------------------------------------------------------------------------
Merged /pig/trunk/src/pig-default.properties:r1593886-1595368
Modified: pig/branches/tez/test/org/apache/pig/parser/TestErrorHandling.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/parser/TestErrorHandling.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/parser/TestErrorHandling.java (original)
+++ pig/branches/tez/test/org/apache/pig/parser/TestErrorHandling.java Sat May 17 00:08:58 2014
@@ -135,7 +135,7 @@ public class TestErrorHandling {
pig.registerQuery( query );
} catch(FrontendException ex) {
System.out.println( ex.getCause().getMessage() );
- Assert.assertTrue( ex.getCause().getMessage().contains( "line 2, column 0" ) );
+ Assert.assertTrue( ex.getMessage().contains( "line 2, column 0" ) );
Assert.assertTrue( ex.getCause().getMessage().contains( "No FileSystem for scheme: fs2you" ) );
return;
}
Modified: pig/branches/tez/test/org/apache/pig/test/TestInputOutputFileValidator.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestInputOutputFileValidator.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestInputOutputFileValidator.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestInputOutputFileValidator.java Sat May 17 00:08:58 2014
@@ -47,7 +47,7 @@ import org.apache.pig.impl.util.LogUtils
import org.apache.pig.newplan.logical.relational.LOLoad;
import org.apache.pig.newplan.logical.relational.LOStore;
import org.apache.pig.newplan.logical.relational.LogicalPlan;
-import org.apache.pig.newplan.logical.rules.InputOutputFileValidator;
+import org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor;
import org.junit.Before;
import org.junit.Test;
@@ -68,8 +68,8 @@ public class TestInputOutputFileValidato
String outputfile = generateNonExistenceTempFile().getAbsolutePath() ;
LogicalPlan plan = genNewLoadStorePlan(inputfile, outputfile, ctx.getFs()) ;
- InputOutputFileValidator executor = new InputOutputFileValidator(plan, ctx) ;
- executor.validate() ;
+ InputOutputFileValidatorVisitor visitor = new InputOutputFileValidatorVisitor(plan, ctx) ;
+ visitor.visit();
}
@Test(expected = VisitorException.class) //should expect an exception
@@ -79,8 +79,8 @@ public class TestInputOutputFileValidato
LogicalPlan plan = genNewLoadStorePlan(inputfile, outputfile, ctx.getDfs()) ;
- InputOutputFileValidator executor = new InputOutputFileValidator(plan, ctx) ;
- executor.validate() ;
+ InputOutputFileValidatorVisitor visitor = new InputOutputFileValidatorVisitor(plan, ctx) ;
+ visitor.visit() ;
}
/**
Modified: pig/branches/tez/test/org/apache/pig/test/TestInputOutputMiniClusterFileValidator.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestInputOutputMiniClusterFileValidator.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestInputOutputMiniClusterFileValidator.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestInputOutputMiniClusterFileValidator.java Sat May 17 00:08:58 2014
@@ -48,7 +48,7 @@ import org.apache.pig.impl.util.LogUtils
import org.apache.pig.newplan.logical.relational.LOLoad;
import org.apache.pig.newplan.logical.relational.LOStore;
import org.apache.pig.newplan.logical.relational.LogicalPlan;
-import org.apache.pig.newplan.logical.rules.InputOutputFileValidator;
+import org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.Test;
@@ -77,8 +77,8 @@ public class TestInputOutputMiniClusterF
LogicalPlan plan = genNewLoadStorePlan(inputfile, outputfile, ctx.getDfs()) ;
- InputOutputFileValidator executor = new InputOutputFileValidator(plan, ctx) ;
- executor.validate() ;
+ InputOutputFileValidatorVisitor executor = new InputOutputFileValidatorVisitor(plan, ctx) ;
+ executor.visit() ;
}
@Test
@@ -88,9 +88,9 @@ public class TestInputOutputMiniClusterF
LogicalPlan plan = genNewLoadStorePlan(inputfile, outputfile, ctx.getDfs()) ;
- InputOutputFileValidator executor = new InputOutputFileValidator(plan, ctx) ;
+ InputOutputFileValidatorVisitor executor = new InputOutputFileValidatorVisitor(plan, ctx) ;
try {
- executor.validate() ;
+ executor.visit() ;
fail("Excepted to fail.");
} catch(Exception e) {
//good
Modified: pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java Sat May 17 00:08:58 2014
@@ -1525,6 +1525,7 @@ public class TestMultiQueryCompiler {
private PhysicalPlan checkPhysicalPlan(LogicalPlan lp, int expectedRoots,
int expectedLeaves, int expectedSize) throws IOException {
+ lp.optimize(myPig.getPigContext());
System.out.println("===== check physical plan =====");
PhysicalPlan pp = ((MRExecutionEngine)myPig.getPigContext().getExecutionEngine()).compile(
Modified: pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java Sat May 17 00:08:58 2014
@@ -622,7 +622,8 @@ public class TestMultiQueryLocal {
private PhysicalPlan checkPhysicalPlan(LogicalPlan lp, int expectedRoots,
int expectedLeaves, int expectedSize) throws IOException {
- System.out.println("===== check physical plan =====");
+ lp.optimize(myPig.getPigContext());
+ System.out.println("===== check physical plan =====");
PhysicalPlan pp = ((MRExecutionEngine)myPig.getPigContext().getExecutionEngine()).compile(
lp, null);
Modified: pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushDownForeachFlatten.java Sat May 17 00:08:58 2014
@@ -17,7 +17,7 @@
*/
package org.apache.pig.test;
-
+import java.io.File;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
@@ -49,6 +49,7 @@ import org.apache.pig.newplan.optimizer.
import org.apache.pig.newplan.optimizer.Rule;
import org.apache.pig.test.utils.Identity;
import org.junit.Assert;
+import org.junit.BeforeClass;
import org.junit.Test;
/**
@@ -57,6 +58,11 @@ import org.junit.Test;
public class TestNewPlanPushDownForeachFlatten {
PigContext pc = new PigContext(ExecType.LOCAL, new Properties());
+ @BeforeClass
+ public static void oneTimeSetup() throws Exception {
+ new File("dummy").delete();
+ }
+
/**
*
* A simple filter UDF for testing
Modified: pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushUpFilter.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushUpFilter.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushUpFilter.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestNewPlanPushUpFilter.java Sat May 17 00:08:58 2014
@@ -17,6 +17,7 @@
*/
package org.apache.pig.test;
+import java.io.File;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
@@ -52,6 +53,7 @@ import org.apache.pig.newplan.optimizer.
import org.apache.pig.newplan.optimizer.Rule;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.BeforeClass;
import org.junit.Test;
/**
@@ -60,6 +62,11 @@ import org.junit.Test;
public class TestNewPlanPushUpFilter {
PigContext pc = new PigContext(ExecType.LOCAL, new Properties());
+ @BeforeClass
+ public static void oneTimeSetup() throws Exception {
+ new File("dummy").delete();
+ }
+
@Before
public void tearDown() {
}
Modified: pig/branches/tez/test/org/apache/pig/test/TestPigStats.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestPigStats.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestPigStats.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestPigStats.java Sat May 17 00:08:58 2014
@@ -166,6 +166,7 @@ public class TestPigStats {
pig.registerQuery("store E into 'alias_output';");
LogicalPlan lp = getLogicalPlan(pig);
+ lp.optimize(pig.getPigContext());
PhysicalPlan pp = ((MRExecutionEngine)pig.getPigContext().getExecutionEngine()).compile(lp,
null);
MROperPlan mp = getMRPlan(pp, pig.getPigContext());
Modified: pig/branches/tez/test/org/apache/pig/test/TestProjectRange.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestProjectRange.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestProjectRange.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestProjectRange.java Sat May 17 00:08:58 2014
@@ -784,13 +784,7 @@ public class TestProjectRange {
private LogicalPlan createAndProcessLPlan(String query) throws FrontendException {
//TODO: create a common util function for logical plan tests
LogicalPlan lp = generateLogicalPlan(query);
- new ColumnAliasConversionVisitor( lp ).visit();
- new SchemaAliasVisitor( lp ).visit();
-
- CompilationMessageCollector collector = new CompilationMessageCollector() ;
- new TypeCheckingRelVisitor( lp, collector).visit();
- new UnionOnSchemaSetter( lp ).visit();
- new CastLineageSetter(lp, collector).visit();
+ lp.validate(pigServer.getPigContext(), "test");
return lp;
Modified: pig/branches/tez/test/org/apache/pig/test/TestStore.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/TestStore.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/TestStore.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/TestStore.java Sat May 17 00:08:58 2014
@@ -26,6 +26,7 @@ import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
@@ -78,7 +79,7 @@ import org.apache.pig.impl.util.Utils;
import org.apache.pig.newplan.Operator;
import org.apache.pig.newplan.logical.relational.LOStore;
import org.apache.pig.newplan.logical.relational.LogicalPlan;
-import org.apache.pig.newplan.logical.rules.InputOutputFileValidator;
+import org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor;
import org.apache.pig.parser.ParserException;
import org.apache.pig.parser.QueryParserDriver;
import org.apache.pig.test.utils.GenRandomData;
@@ -149,7 +150,6 @@ public class TestStore {
"i:int,d:double);" +
"store a into '" + outputFileName + "' using " + "PigStorage();";
org.apache.pig.newplan.logical.relational.LogicalPlan lp = Util.buildLp( pig, query );
- new InputOutputFileValidator(lp, pig.getPigContext()).validate();
} catch (PlanValidationException e){
// Since output file is not present, validation should pass
// and not throw this exception.
@@ -169,9 +169,9 @@ public class TestStore {
String query = "a = load '" + inputFileName + "' as (c:chararray, " +
"i:int,d:double);" +
"store a into '" + outputFileName + "' using PigStorage();";
- org.apache.pig.newplan.logical.relational.LogicalPlan lp = Util.buildLp( pig, query );
- new InputOutputFileValidator(lp, pig.getPigContext()).validate();
- } catch (FrontendException pve){
+ Util.buildLp( pig, query );
+ } catch (InvocationTargetException e){
+ FrontendException pve = (FrontendException)e.getCause();
// Since output file is present, validation should fail
// and throw this exception
assertEquals(6000,pve.getErrorCode());
Modified: pig/branches/tez/test/org/apache/pig/test/Util.java
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/Util.java?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/Util.java (original)
+++ pig/branches/tez/test/org/apache/pig/test/Util.java Sat May 17 00:08:58 2014
@@ -65,6 +65,7 @@ import org.apache.pig.backend.executione
import org.apache.pig.backend.hadoop.datastorage.ConfigurationUtil;
import org.apache.pig.backend.hadoop.executionengine.HExecutionEngine;
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler;
+import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRExecutionEngine;
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher;
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.plans.MROperPlan;
import org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhysicalPlan;
@@ -84,7 +85,6 @@ import org.apache.pig.impl.logicalLayer.
import org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema;
import org.apache.pig.impl.plan.CompilationMessageCollector;
import org.apache.pig.impl.util.LogUtils;
-import org.apache.pig.newplan.logical.optimizer.DanglingNestedNodeRemover;
import org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter;
import org.apache.pig.newplan.logical.optimizer.SchemaResetter;
import org.apache.pig.newplan.logical.optimizer.UidResetter;
@@ -94,6 +94,7 @@ import org.apache.pig.newplan.logical.re
import org.apache.pig.newplan.logical.relational.LogicalSchema.LogicalFieldSchema;
import org.apache.pig.newplan.logical.visitor.CastLineageSetter;
import org.apache.pig.newplan.logical.visitor.ColumnAliasConversionVisitor;
+import org.apache.pig.newplan.logical.visitor.DanglingNestedNodeRemover;
import org.apache.pig.newplan.logical.visitor.ScalarVisitor;
import org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor;
import org.apache.pig.newplan.logical.visitor.SortInfoSetter;
@@ -1066,7 +1067,8 @@ public class Util {
public static PhysicalPlan buildPp(PigServer pigServer, String query)
throws Exception {
LogicalPlan lp = buildLp( pigServer, query );
- return ((HExecutionEngine)pigServer.getPigContext().getExecutionEngine()).compile(lp,
+ lp.optimize(pigServer.getPigContext());
+ return ((MRExecutionEngine)pigServer.getPigContext().getExecutionEngine()).compile(lp,
pigServer.getPigContext().getProperties());
}
@@ -1075,16 +1077,7 @@ public class Util {
QueryParserDriver parserDriver = new QueryParserDriver( pc, "test", fileNameMap );
org.apache.pig.newplan.logical.relational.LogicalPlan lp = parserDriver.parse( query );
- new ColumnAliasConversionVisitor(lp).visit();
- new SchemaAliasVisitor(lp).visit();
- new ScalarVisitor(lp, pc, "test").visit();
-
- CompilationMessageCollector collector = new CompilationMessageCollector() ;
-
- new TypeCheckingRelVisitor( lp, collector).visit();
-
- new UnionOnSchemaSetter( lp ).visit();
- new CastLineageSetter(lp, collector).visit();
+ lp.validate(pc, "test");
return lp;
}
@@ -1093,16 +1086,7 @@ public class Util {
QueryParserDriver parserDriver = new QueryParserDriver( pc, "test", fileNameMap );
org.apache.pig.newplan.logical.relational.LogicalPlan lp = parserDriver.parse( query );
- new ColumnAliasConversionVisitor( lp ).visit();
- new SchemaAliasVisitor( lp ).visit();
- new ScalarVisitor(lp, pc, "test").visit();
-
- CompilationMessageCollector collector = new CompilationMessageCollector() ;
-
- new TypeCheckingRelVisitor( lp, collector).visit();
-
- new UnionOnSchemaSetter( lp ).visit();
- new CastLineageSetter(lp, collector).visit();
+ lp.validate(pc, "test");
return lp;
}
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Arithmetic.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Arithmetic.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Arithmetic.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Arithmetic.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-12: Store()
+A: Store()
|
|---A: New For Each(false,false,false)[bag] - scope-64
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/BinCond.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/BinCond.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/BinCond.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/BinCond.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-16: Store()
+A: Store()
|
|---A: New For Each(false)[bag] - scope-106
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-10: Store()
+A: Store()
|
|---A: Package(Packager)[tuple]{tuple} - scope-19
|
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Comparison.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Comparison.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Comparison.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Comparison.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-14: Store()
+A: Store()
|
|---A: Filter[bag] - scope-67
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/ComplexForeach.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/ComplexForeach.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/ComplexForeach.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/ComplexForeach.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-1: Store()
+C: Store()
|
|---C: New For Each(false)[bag] - scope-7
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Generate.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Generate.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Generate.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Generate.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-18: Store()
+A: Store()
|
|---A: New For Each(false,false,false,false,false,false)[bag] - scope-148
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/IsNull1.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/IsNull1.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/IsNull1.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/IsNull1.gld Sat May 17 00:08:58 2014
@@ -10,7 +10,7 @@ y: Filter[bag] - scope-170
|
|---test-26: Load()
-test-28: Store()
+x: Store()
|
|---x: Filter[bag] - scope-174
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Split1.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Split1.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Split1.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Split1.gld Sat May 17 00:08:58 2014
@@ -12,7 +12,7 @@ y: Filter[bag] - scope-157
|
|---test-23: Load()
-test-25: Store()
+x: Store()
|
|---x: Filter[bag] - scope-162
| |
Modified: pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Union.gld
URL: http://svn.apache.org/viewvc/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Union.gld?rev=1595392&r1=1595391&r2=1595392&view=diff
==============================================================================
--- pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Union.gld (original)
+++ pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/Union.gld Sat May 17 00:08:58 2014
@@ -1,4 +1,4 @@
-test-22: Store()
+A: Store()
|
|---A: Union[bag] - scope-153
|
Propchange: pig/branches/tez/test/org/apache/pig/test/data/bzipdir1.bz2/bzipdir2.bz2/recordLossblockHeaderEndsAt136500.txt.bz2
------------------------------------------------------------------------------
Merged /pig/trunk/test/org/apache/pig/test/data/bzipdir1.bz2/bzipdir2.bz2/recordLossblockHeaderEndsAt136500.txt.bz2:r1593886-1595368