You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "fang fang chen (JIRA)" <ji...@apache.org> on 2011/09/13 14:47:08 UTC
[jira] [Created] (PIG-2281) pig-0.8.1: some unit test case failed
with open source JDK
pig-0.8.1: some unit test case failed with open source JDK
----------------------------------------------------------
Key: PIG-2281
URL: https://issues.apache.org/jira/browse/PIG-2281
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.1
Environment: open source jdk
ant: 1.8.2
Reporter: fang fang chen
Fix For: 0.8.1
Based on my analysis, I think this is because HashMap does not guarantee ordering.
1.
[junit] Running org.apache.pig.test.TestDataModel
[junit] Tests run: 22, Failures: 1, Errors: 0, Time elapsed: 0.382 sec
Caused by different output of HashMap.toString() with open source JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in open source jdk is also correct.
Detail:
Testcase: testTupleToString took 0.002 sec
FAILED
toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
public void testTupleToString() throws Exception {
Tuple t = giveMeOneOfEach();
assertEquals("toString", "((3,3.0),{(4),(mary had a little lamb)},[hello#world,goodbye#all],42,5000000000,3.1415927,2.99792458E8,true,hello,goodbye,)", t.toString());//line 269
}
comment:
private Tuple giveMeOneOfEach() throws Exception {
TupleFactory tf = TupleFactory.getInstance();
Tuple t1 = tf.newTuple(11);
Tuple t2 = tf.newTuple(2);
t2.set(0, new Integer(3));
t2.set(1, new Float(3.0));
DataBag bag = BagFactory.getInstance().newDefaultBag();
bag.add(tf.newTuple(new Integer(4)));
bag.add(tf.newTuple(new String("mary had a little lamb")));
Map<String, Object> map = new HashMap<String, Object>(2);
map.put(new String("hello"), new String("world"));
map.put(new String("goodbye"), new String("all"));
t1.set(0, t2);
t1.set(1, bag);
t1.set(2, map);//when run t.toString(), HashMap.toString() will be invoked
t1.set(3, new Integer(42));
t1.set(4, new Long(5000000000L));
t1.set(5, new Float(3.141592654));
t1.set(6, new Double(2.99792458e8));
t1.set(7, new Boolean(true));
t1.set(8, new DataByteArray("hello"));
t1.set(9, new String("goodbye"));
return t1;
}
}
2.
[junit] Running org.apache.pig.test.TestLogToPhyCompiler
[junit] Tests run: 23, Failures: 1, Errors: 0, Time elapsed: 1.16 sec
Maybe caused by different output of HashMap.keySet() with open source JDK and sun jdk.
Detail:
Failure information:
Testcase: testSplit took 0.226 sec
FAILED
Plan not match
junit.framework.AssertionFailedError: Plan not match
at org.apache.pig.test.TestLogToPhyCompiler.testSplit(TestLogToPhyCompiler.java:444)
public void testSplit() throws VisitorException, IOException {
String query = "split (load 'a') into x if $0 < '7', y if $0 > '7';";
LogicalPlan plan = buildPlan(query);
log.info("ff test plan:"+plan);
PhysicalPlan pp = buildPhysicalPlan(plan);
log.info("ff test pp:"+pp);
int MAX_SIZE = 100000;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
pp.explain(baos);
baos.write((int)'\n');
String compiledPlan = baos.toString();
compiledPlan = compiledPlan.replaceAll("Load(.*)","Load()");
if(generate){
FileOutputStream fos = new FileOutputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
fos.write(baos.toByteArray());
return;
}
FileInputStream fis1 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
FileInputStream fis2 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split2.gld");
byte[] b1 = new byte[MAX_SIZE];
byte[] b2 = new byte[MAX_SIZE];
int len = fis1.read(b1);
int test = fis2.read(b2);
//System.out.println("Length of first plan = " + len + " of second = " + test);
String goldenPlan1 = new String(b1, 0, len);
String goldenPlan2 = new String(b2, 0, len);
goldenPlan1 = goldenPlan1.replaceAll("Load(.*)","Load()");
goldenPlan2 = goldenPlan2.replaceAll("Load(.*)","Load()");
System.out.println();
System.out.println(compiledPlan);
System.out.println("-------------");
if(compiledPlan.compareTo(goldenPlan1) == 0 || compiledPlan.compareTo(goldenPlan2) == 0) {
// good
}
else {
System.out.println("Expected plan1=") ;
System.out.println(goldenPlan1) ;
System.out.println("Expected plan2=") ;
System.out.println(goldenPlan1) ;
System.out.println("Actual plan=") ;
System.out.println(compiledPlan) ;
System.out.println("**END**") ;
fail("Plan not match") ;//line 444
}
}
comment:
variable compiledPlan initialize invoke the following methods:
pp.explain(baos);// explain(OutputStream out) method
baos.write((int)'\n');
String compiledPlan = baos.toString();
explain(OutputStream out)explain(OutputStream out, boolean verbose)print(OutputStream printer)depthFirstPP()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
baos variable output:
> open source jdk:
> x: Filter[tuple] - Test-Plan-Builder-240
> | |
> | Less Than[boolean] - Test-Plan-Builder-243
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-241
> | |
> | |---Constant(7) - Test-Plan-Builder-242
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
>
> y: Filter[tuple] - Test-Plan-Builder-244
> | |
> | Greater Than[boolean] - Test-Plan-Builder-247
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-245
> | |
> | |---Constant(7) - Test-Plan-Builder-246
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
> sun jdk:
> y: Filter[tuple] - Test-Plan-Builder-240
> | |
> | Greater Than[boolean] - Test-Plan-Builder-243
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-241
> | |
> | |---Constant(7) - Test-Plan-Builder-242
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
>
> x: Filter[tuple] - Test-Plan-Builder-244
> | |
> | Less Than[boolean] - Test-Plan-Builder-247
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-245
> | |
> | |---Constant(7) - Test-Plan-Builder-246
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
3.
[junit] Running org.apache.pig.test.TestMRCompiler
[junit] Tests run: 25, Failures: 1, Errors: 0, Time elapsed: 0.729 sec
Maybe, caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
Detail:
Testcase: testSortUDF1 took 0.02 sec
FAILED
null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
| ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
| ...>
junit.framework.ComparisonFailure: null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
| ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
| ...>
at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
private void run(PhysicalPlan pp, String expectedFile) throws Exception {
String compiledPlan, goldenPlan = null;
int MAX_SIZE = 100000;
MRCompiler comp = new MRCompiler(pp, pc);
comp.compile();
MROperPlan mrp = comp.getMRPlan();
PlanPrinter ppp = new PlanPrinter(mrp);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ppp.print(baos);//see "comments"
compiledPlan = baos.toString();//compiledPlan's initialize is based on baos
if(generate ){
FileOutputStream fos = new FileOutputStream(expectedFile);
fos.write(baos.toByteArray());
return;
}
FileInputStream fis = new FileInputStream(expectedFile);
byte[] b = new byte[MAX_SIZE];
int len = fis.read(b);
goldenPlan = new String(b, 0, len);
if (goldenPlan.charAt(len-1) == '\n')
goldenPlan = goldenPlan.substring(0, len-1);
pp.explain(System.out);
System.out.println();
System.out.println("<<<" + compiledPlan + ">>>");
System.out.println("-------------");
System.out.println("Golden");
System.out.println("<<<" + goldenPlan + ">>>");
System.out.println("-------------");
assertEquals(goldenPlan, compiledPlan);//line 1056
}
comment:
ppp.print(baos) invokes method as following:
print(OutputStream printer) depthFirstPP()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
4.
[junit] Running org.apache.pig.test.TestMergeJoinOuter
[junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 132.66 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
Testcase: testCompilation took 0.443 sec
FAILED
junit.framework.AssertionFailedError:
at org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116)
Iterator<MapReduceOper> itr = mrPlan.iterator();// see comments
MapReduceOper oper = itr.next();
assertTrue(oper.reducePlan.isEmpty());//line 116
comments:
iterator() method:
public Iterator<E> iterator() {
return mOps.keySet().iterator();//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
}
with same mrPlan:
MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
Reduce Plan Empty
| C: Store(hdfs://localhost.localdomain:34390/user/root/out:org.apache.pig.builtin.PigStorage) - scope-38
| |
| |---C: New For Each(true,true)[tuple] - scope-37
| | |
| | Project[bag][1] - scope-31
| | |
| | POBinCond[bag] - scope-36
| | |
| | |---Project[bag][2] - scope-32
| | |
| | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - scope-34
| | | |
| | | |---Project[bag][2] - scope-33
| | |
| | |---Constant({(,,)}) - scope-35
| |
| |---C: MergeCogroup[tuple] - scope-30
| |
| |---A: Load(hdfs://localhost.localdomain:34390/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader) - scope-22
|
|---MapReduce(1) - scope-41:
| Store(hdfs://localhost.localdomain:34390/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage) - scope-48
| |
| |---New For Each(true)[bag] - scope-47
| | |
| | Project[tuple][1] - scope-46
| |
| |---Package[tuple]{tuple} - scope-45
| Local Rearrange[tuple]{tuple}(false) - scope-44
| | |
| | Project[tuple][*] - scope-43
| |
variable itr is in different order:
open source jdk:
(Name: MapReduce(1) - scope-41(itr.next())
(Name: MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
sun jdk:
(Name: MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) - scope-39: (itr.next())
(Name: MapReduce(1) - scope-41:
5.
[junit] Running org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor
[junit] Tests run: 25, Failures: 3, Errors: 0, Time elapsed: 1.081 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
(1).
Testcase: testSimplePlan took 0.295 sec
FAILED
expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testSimplePlan(TestNewPlanLogToPhyTranslationVisitor.java:131)
public void testSimplePlan() throws Exception {
LogicalPlanTester lpt = new LogicalPlanTester(pc);
lpt.buildPlan("a = load 'd.txt';");
lpt.buildPlan("b = filter a by $0==NULL;");
LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
org.apache.pig.newplan.logical.relational.LogicalPlan newLogicalPlan = migratePlan(plan);
PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
assertEquals( 3, phyPlan.size() );
assertEquals( 1, phyPlan.getRoots().size() );
assertEquals( 1, phyPlan.getLeaves().size() );
PhysicalOperator load = phyPlan.getRoots().get(0);
assertEquals( POLoad.class, load.getClass() );
assertTrue( ((POLoad)load).getLFile().getFileName().contains("d.txt") );
// Check for Filter
PhysicalOperator fil = phyPlan.getSuccessors(load).get(0);
assertEquals( POFilter.class, fil.getClass() );
PhysicalPlan filPlan = ((POFilter)fil).getPlan();
assertEquals( 2, filPlan.getRoots().size() );
assertEquals( 1, filPlan.getLeaves().size() );
PhysicalOperator eq = filPlan.getLeaves().get(0);
assertEquals( EqualToExpr.class, eq.getClass() );
PhysicalOperator prj1 = filPlan.getRoots().get(0);
assertEquals( POProject.class, prj1.getClass() );//line 131
assertEquals( 0, ((POProject)prj1).getColumn() );
PhysicalOperator constExp = filPlan.getRoots().get(1);
assertEquals( ConstantExpression.class, constExp.getClass() );
assertEquals( null, ((ConstantExpression)constExp).getValue() );
// Check for Store
PhysicalOperator stor = phyPlan.getSuccessors(fil).get(0);
assertEquals( POStore.class, stor.getClass() );
assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty"));
}
comment:
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
(2).
Testcase: testJoinPlan took 0.062 sec
FAILED
expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testJoinPlan(TestNewPlanLogToPhyTranslationVisitor.java:201)
public void testJoinPlan() throws Exception {
LogicalPlanTester lpt = new LogicalPlanTester(pc);
lpt.buildPlan("a = load 'd1.txt' as (id, c);");
lpt.buildPlan("b = load 'd2.txt'as (id, c);");
lpt.buildPlan("c = join a by id, b by c;");
lpt.buildPlan("d = filter c by a::id==NULL AND b::c==NULL;");
LogicalPlan plan = lpt.buildPlan("store d into 'empty';");
// check basics
org.apache.pig.newplan.logical.relational.LogicalPlan newPlan = migratePlan(plan);
PhysicalPlan physicalPlan = translatePlan(newPlan);
assertEquals(9, physicalPlan.size());
assertEquals(physicalPlan.getRoots().size(), 2);
// Check Load and LocalRearrange and GlobalRearrange
PhysicalOperator LoR = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(0)).get(0);
assertEquals( POLocalRearrange.class, LoR.getClass() );
POLocalRearrange Lor = (POLocalRearrange) LoR;
PhysicalOperator prj3 = Lor.getPlans().get(0).getLeaves().get(0);
assertEquals( POProject.class, prj3.getClass() );
assertEquals(0, ((POProject)prj3).getColumn() );
PhysicalOperator inp1 = Lor.getInputs().get(0);
assertEquals( POLoad.class, inp1.getClass() );
assertTrue( ((POLoad)inp1).getLFile().getFileName().contains("d1.txt") );
PhysicalOperator LoR1 = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(1)).get(0);
assertEquals( POLocalRearrange.class, LoR1.getClass() );
POLocalRearrange Lor1 = (POLocalRearrange) LoR1;
PhysicalOperator prj4 = Lor1.getPlans().get(0).getLeaves().get(0);
assertEquals( POProject.class, prj4.getClass() );
assertEquals(1, ((POProject)prj4).getColumn() );
PhysicalOperator inp2 = Lor1.getInputs().get(0);
assertEquals( POLoad.class, inp2.getClass() );
assertTrue( ((POLoad)inp2).getLFile().getFileName().contains("d2.txt") );
PhysicalOperator GoR = (PhysicalOperator)physicalPlan.getSuccessors(LoR).get(0);
assertEquals( POGlobalRearrange.class, GoR.getClass() );
PhysicalOperator Pack = (PhysicalOperator)physicalPlan.getSuccessors(GoR).get(0);
assertEquals( POPackage.class, Pack.getClass() );
// Check for ForEach
PhysicalOperator ForE = (PhysicalOperator)physicalPlan.getSuccessors(Pack).get(0);
assertEquals( POForEach.class, ForE.getClass() );
PhysicalOperator prj5 = ((POForEach)ForE).getInputPlans().get(0).getLeaves().get(0);
assertEquals( POProject.class, prj5.getClass() );
assertEquals( 1, ((POProject)prj5).getColumn() );
PhysicalOperator prj6 = ((POForEach)ForE).getInputPlans().get(1).getLeaves().get(0);
assertEquals( POProject.class, prj6.getClass() );
assertEquals( 2, ((POProject)prj6).getColumn() );
// Filter Operator
PhysicalOperator fil = (PhysicalOperator)physicalPlan.getSuccessors(ForE).get(0);
assertEquals( POFilter.class, fil.getClass() );
PhysicalPlan filPlan = ((POFilter)fil).getPlan();
List<PhysicalOperator> filRoots = filPlan.getRoots();
assertEquals( ConstantExpression.class, filRoots.get(1).getClass() );//line 201
ConstantExpression ce1 = (ConstantExpression) filRoots.get(1);
assertEquals( null, ce1.getValue() );
assertEquals( ConstantExpression.class, filRoots.get(3).getClass() );
ConstantExpression ce2 = (ConstantExpression) filRoots.get(3);
assertEquals( null, ce2.getValue() );
assertEquals( POProject.class, filRoots.get(0).getClass() );
POProject prj1 = (POProject) filRoots.get(0);
assertEquals( 3, prj1.getColumn() );
assertEquals( POProject.class, filRoots.get(2).getClass() );
POProject prj2 = (POProject) filRoots.get(2);
assertEquals( 0, prj2.getColumn() );
// Check Store Operator
PhysicalOperator stor = (PhysicalOperator)physicalPlan.getSuccessors(fil).get(0);
assertEquals( POStore.class, stor.getClass() );
assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty") );
}
comment:
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
}
(3).
Testcase: testMultiStore took 0.083 sec
FAILED
expected:<1> but was:<0>
junit.framework.AssertionFailedError: expected:<1> but was:<0>
at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testMultiStore(TestNewPlanLogToPhyTranslationVisitor.java:255)
PhysicalOperator prj2 = Lor1.getPlans().get(0).getLeaves().get(0);
assertEquals(1, ((POProject)prj2).getColumn() );//line 255
comment:
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
6.
[junit] Running org.apache.pig.test.TestPruneColumn
[junit] Tests run: 67, Failures: 4, Errors: 0, Time elapsed: 528.047 sec
Maybe caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
(1).
Testcase: testMapKey2 took 6.291 sec
FAILED
null
junit.framework.AssertionFailedError: null
at org.apache.pig.test.TestPruneColumn.testMapKey2(TestPruneColumn.java:1213)
public void testMapKey2() throws Exception{
pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");// see comment3
pigServer.registerQuery("C = foreach B generate $0#'key2', $1;");
Iterator<Tuple> iter = pigServer.openIterator("C");//see comment1
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("2"));
assertTrue(t.get(1).toString().equals("1"));
assertTrue(iter.hasNext());
t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("4"));
assertTrue(t.get(1).toString().equals("2"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0",
"Map key required for A: $1->[key2, key1]"}));// line 1213 see comment2
}
comments1:
pigServer.openIterator("C") invokes the following method to save pigServer information to filename:
store(String id, String filename, String func)
comment2:
public boolean checkLogFileMessage(String[] messages)
{
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(logFile));//logFile=filename
List<String> logMessages=new ArrayList<String>();
String line;
while ((line=reader.readLine())!=null)
{
logMessages.add(line);
}
// Check if all messages appear in the log
for (int i=0;i<messages.length;i++)
{
boolean found = false;
for (int j=0;j<logMessages.size();j++)
if (logMessages.get(j).contains(messages[i])) {
found = true;
break;
}
if (!found)
return false;
}
// Check no other log besides messages
for (int i=0;i<logMessages.size();i++) {
boolean found = false;
for (int j=0;j<messages.length;j++) {
if (logMessages.get(i).contains(messages[j])) {
found = true;
break;
}
}
if (!found) {
if (logMessages.get(i).contains("Columns pruned for")||
logMessages.get(i).contains("Map key required for")) {
return false;
}
}
}
return true;
}
catch (IOException e) {
return false;
}
}
comment3:
the content in filename is different, because pigServer(invoke HashMap.keySet()) is different.
pigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
(2)
Testcase: testMapKey3 took 6.319 sec
FAILED
null
junit.framework.AssertionFailedError: null
at org.apache.pig.test.TestPruneColumn.testMapKey3(TestPruneColumn.java:1229)
public void testMapKey3() throws Exception {
pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");
pigServer.registerQuery("C = group B all;");
Iterator<Tuple> iter = pigServer.openIterator("C");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("all"));
assertTrue(t.get(1).toString().equals("{([key2#2,key1#1],1),([key2#4,key1#2],2)}"));
//line 1229
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0"}));
}
comment:
variable "t" initialize process:
titerpigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
different t.get(1).toString():
OPEN SOURCE JDK:
{([key1#1,key2#2],1),([key1#2,key2#4],2)}
SUN JDK:
{([key2#2,key1#1],1),([key2#4,key1#2],2)}
(3).
Testcase: testMapKeyInSplit1 took 6.3 sec
FAILED
null
junit.framework.AssertionFailedError: null
at org.apache.pig.test.TestPruneColumn.testMapKeyInSplit1(TestPruneColumn.java:1303)
public void testMapKeyInSplit1() throws Exception {
pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as (m:map[]);");
pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
pigServer.registerQuery("D = join B by key1, C by key2;");
Iterator<Tuple> iter = pigServer.openIterator("D");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("2"));
assertTrue(t.get(1).toString().equals("2"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Map key required for A: $0->[key2, key1]"}));//line 1303
}
comment: same with (1).
(4).
Testcase: testSharedSchemaObject took 6.327 sec
FAILED
null
junit.framework.AssertionFailedError: null
at org.apache.pig.test.TestPruneColumn.testSharedSchemaObject(TestPruneColumn.java:1626)
public void testSharedSchemaObject() throws Exception {
pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile10.toString(), pigServer.getPigContext()) + "' AS (a0, a1:map[], a2);");
pigServer.registerQuery("B = foreach A generate a1;");
pigServer.registerQuery("C = limit B 10;");
Iterator<Tuple> iter = pigServer.openIterator("C");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.toString().equals("([2#1,1#1])"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0, $2"}));// line 1626
}
comment: same with (2).
7.
[junit] Running org.apache.pig.test.TestUnionOnSchema
[junit] Tests run: 21, Failures: 1, Errors: 0, Time elapsed: 196.841 sec
Testcase: testUnionOnSchemaScopedColumnNameNeg took 0.008 sec
FAILED
Expected exception message matching 'Found more than one match: l1::i, l2::i' but got 'Error during parsing. Found more than one match: l2::i, l1::i'
at org.apache.pig.test.TestUnionOnSchema.checkSchemaEx(TestUnionOnSchema.java:604)
at org.apache.pig.test.TestUnionOnSchema.testUnionOnSchemaScopedColumnNameNeg(TestUnionOnSchema.java:370)
8..
[junit] Running org.apache.pig.test.TestPushDownForeachFlatten
[junit] Tests run: 37, Failures: 0, Errors: 8, Time elapsed: 1.455 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
(1)
Testcase: testForeachUnion took 0.039 sec
Caused an ERROR
Expected LOForEach, got LOUnion
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOUnion
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachUnion(TestPushDownForeachFlatten.java:275)
public void testForeachUnion() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
LogicalPlan lp = planTester.buildPlan("D = union B, C;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);// see comment
assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 275
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment:
LOLoad load = (LOLoad) lp.getRoots().get(0);//
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
}
(2)
Testcase: testForeachCogroup took 0.038 sec
Caused an ERROR
Expected LOForEach, got LOCogroup
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCogroup
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCogroup(TestPushDownForeachFlatten.java:295)
public void testForeachCogroup() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
LogicalPlan lp = planTester.buildPlan("D = cogroup B by $0, C by $0;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 295
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment: same with (1)
(3)
Testcase: testForeachCross took 0.035 sec
Caused an ERROR
Expected LOForEach, got LOCross
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCross(TestPushDownForeachFlatten.java:427)
public void testForeachCross() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
planTester.buildPlan("D = cross B, C;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
LOLimit limit = (LOLimit) lp.getLeaves().get(0);
LOCross cross = (LOCross)lp.getPredecessors(limit).get(0);
LOForEach foreach = (LOForEach) lp.getPredecessors(cross).get(0);
Schema limitSchema = limit.getSchema();
assertTrue(pushDownForeach.check(lp.getSuccessors(load)));
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == true);
assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
pushDownForeach.transform(lp.getSuccessors(load));//line 427
planTester.rebuildSchema(lp);
for(Boolean b: foreach.getFlatten()) {
assertEquals(b.booleanValue(), false);
}
LOForEach newForeach = (LOForEach)lp.getSuccessors(cross).get(0);
List<Boolean> newForeachFlatten = newForeach.getFlatten();
Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
for(Integer key: remap.keySet()) {
Integer value = remap.get(key);
assertEquals(newForeachFlatten.get(value).booleanValue(), true);
}
assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
}
comment: same with (1)
(4)
Testcase: testForeachFlattenAddedColumnCross took 0.034 sec
Caused an ERROR
Expected LOForEach, got LOCross
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnCross(TestPushDownForeachFlatten.java:545)
public void testForeachFlattenAddedColumnCross() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
planTester.buildPlan("D = cross B, C;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad loada = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 545
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment: same with (1)
(5)
Testcase: testForeachFRJoin took 0.027 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFRJoin(TestPushDownForeachFlatten.java:619)
public void testForeachFRJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
LOLimit limit = (LOLimit) lp.getLeaves().get(0);
LOJoin frjoin = (LOJoin)lp.getPredecessors(limit).get(0);
LOForEach foreach = (LOForEach) lp.getPredecessors(frjoin).get(0);
Schema limitSchema = limit.getSchema();
assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 619
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == true);
assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
pushDownForeach.transform(lp.getSuccessors(load));
planTester.rebuildSchema(lp);
for(Boolean b: foreach.getFlatten()) {
assertEquals(b.booleanValue(), false);
}
LOForEach newForeach = (LOForEach)lp.getSuccessors(frjoin).get(0);
List<Boolean> newForeachFlatten = newForeach.getFlatten();
Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
for(Integer key: remap.keySet()) {
Integer value = remap.get(key);
assertEquals(newForeachFlatten.get(value).booleanValue(), true);
}
assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
}
comment: same with (1)
(6)
Testcase: testForeachFlattenAddedColumnFRJoin took 0.026 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnFRJoin(TestPushDownForeachFlatten.java:738)
public void testForeachFlattenAddedColumnFRJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad loada = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 738
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment: same with (1)
(7)
Testcase: testForeachInnerJoin took 0.026 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachInnerJoin(TestPushDownForeachFlatten.java:812)
public void testForeachInnerJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
planTester.buildPlan("D = join B by $0, C by $0;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
LOLimit limit = (LOLimit) lp.getLeaves().get(0);
LOJoin join = (LOJoin)lp.getPredecessors(limit).get(0);
LOForEach foreach = (LOForEach) lp.getPredecessors(join).get(0);
Schema limitSchema = limit.getSchema();
assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 812
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == true);
assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
pushDownForeach.transform(lp.getSuccessors(load));
planTester.rebuildSchema(lp);
for(Boolean b: foreach.getFlatten()) {
assertEquals(b.booleanValue(), false);
}
LOForEach newForeach = (LOForEach)lp.getSuccessors(join).get(0);
List<Boolean> newForeachFlatten = newForeach.getFlatten();
Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
for(Integer key: remap.keySet()) {
Integer value = remap.get(key);
assertEquals(newForeachFlatten.get(value).booleanValue(), true);
}
assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
}
comment: same with (1)
(8)
Testcase: testForeachFlattenAddedColumnInnerJoin took 0.021 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnInnerJoin(TestPushDownForeachFlatten.java:931)
public void testForeachFlattenAddedColumnInnerJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
planTester.buildPlan("D = join B by $0, C by $0;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad loada = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 931
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
9.
[junit] Running org.apache.pig.test.TestTypeCheckingValidator
[junit] Tests run: 120, Failures: 0, Errors: 1, Time elapsed: 15.047 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in OPEN SOURCE jdk is also correct.
Detail:
Testcase: testMapLookupLineage took 0.012 sec
Caused an ERROR
org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
java.lang.ClassCastException: org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
at org.apache.pig.test.TestTypeCheckingValidator.testMapLookupLineage(TestTypeCheckingValidator.java:5397)
public void testMapLookupLineage() throws Throwable {
planTester.buildPlan("a = load 'a' using BinStorage() as (field1, field2: float, field3: chararray );") ;
planTester.buildPlan("b = foreach a generate field1#'key1' as map1;") ;
LogicalPlan plan = planTester.buildPlan("c = foreach b generate map1#'key2' + 1 ;") ;
// validate
CompilationMessageCollector collector = new CompilationMessageCollector() ;
TypeCheckingValidator typeValidator = new TypeCheckingValidator() ;
typeValidator.validate(plan, collector) ;
printMessageCollector(collector) ;
printTypeGraph(plan) ;
planTester.printPlan(plan, TypeCheckingTestUtil.getCurrentMethodName());
if (collector.hasError()) {
throw new AssertionError("Expect no error") ;
}
LOForEach foreach = (LOForEach)plan.getLeaves().get(0);// see comement1
LogicalPlan foreachPlan = foreach.getForEachPlans().get(0);
LogicalOperator exOp = foreachPlan.getRoots().get(0);//see comment2
// the root would be the project and there would be cast
// to map between the project and LOMapLookup
LOCast cast1 = (LOCast)foreachPlan.getSuccessors(exOp).get(0);//line 5397
assertTrue(cast1.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
LOMapLookup map = (LOMapLookup)foreachPlan.getSuccessors(cast1).get(0);
LOCast cast = (LOCast)foreachPlan.getSuccessors(map).get(0);
assertTrue(cast.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
}
comment1:
foreachPlan initialize process, this will cause the different foreachPlan.
foreachPlanforeachplan.getLeaves
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
comment2:
exOpforeachPlan.getRoots(), this is the root cause
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
}
SUN JDK output:
foreachPlan:
Add scope-12 FieldSchema: int Type: int
|
|---Const scope-15( 1 ) FieldSchema: int Type: int
|
|---Cast scope-19 FieldSchema: int Type: int
|
|---MapLookup scope-14 FieldSchema: bytearray Type: bytearray
|
|---Cast scope-18 FieldSchema: map Type: map
|
|---Project scope-13 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
Input: b: ForEach scope-6
getRoots method process:
keySet() method:
Cast scope-19
Cast scope-18
Add scope-12
Project scope-13 Projections: [0] Overloaded: false(add to mRoots)
MapLookup scope-14
Const scope-15( 1 )(add to mRoots)
foreachPlan.getRoots():(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
(Name: Const scope-15( 1 ) Operator Key: scope-15)
exOp:(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
OPEN SOURCE JDK output:
foreachPlan:
Add scope-13 FieldSchema: int Type: int
|
|---Const scope-12( 1 ) FieldSchema: int Type: int
|
|---Cast scope-19 FieldSchema: int Type: int
|
|---MapLookup scope-15 FieldSchema: bytearray Type: bytearray
|
|---Cast scope-18 FieldSchema: map Type: map
|
|---Project scope-14 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
Input: b: ForEach scope-6
getRoots method process:
keySet() method:
Cast scope-18
Cast scope-19
Const scope-12( 1 )(add to mRoots)
Add scope-13
Project scope-14 Projections: [0] Overloaded: false(add to mRoots)
MapLookup scope-15
foreachPlan.getRoots():(Name: Const scope-12( 1 ) Operator Key: scope-12)
(Name: Project scope-14 Projections: [0] Overloaded: false Operator Key: scope-14)//output in different order compared with SUN JDK output
exOp:(Name: Const scope-12( 1 ) Operator Key: scope-12)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2281) pig-0.8.1: some unit test case failed
with open source JDK
Posted by "fang fang chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
fang fang chen resolved PIG-2281.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.11
Duplicated with PIG-2405
> pig-0.8.1: some unit test case failed with open source JDK
> ----------------------------------------------------------
>
> Key: PIG-2281
> URL: https://issues.apache.org/jira/browse/PIG-2281
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: open source jdk
> ant: 1.8.2
> Reporter: fang fang chen
> Assignee: fang fang chen
> Fix For: 0.11, 0.8.1
>
>
> Based on my analysis, I think this is because HashMap does not guarantee ordering.
> 1.
> [junit] Running org.apache.pig.test.TestDataModel
> [junit] Tests run: 22, Failures: 1, Errors: 0, Time elapsed: 0.382 sec
> Caused by different output of HashMap.toString() with open source JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in open source jdk is also correct.
> Detail:
> Testcase: testTupleToString took 0.002 sec
> FAILED
> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
> public void testTupleToString() throws Exception {
> Tuple t = giveMeOneOfEach();
> assertEquals("toString", "((3,3.0),{(4),(mary had a little lamb)},[hello#world,goodbye#all],42,5000000000,3.1415927,2.99792458E8,true,hello,goodbye,)", t.toString());//line 269
> }
> comment:
> private Tuple giveMeOneOfEach() throws Exception {
> TupleFactory tf = TupleFactory.getInstance();
> Tuple t1 = tf.newTuple(11);
> Tuple t2 = tf.newTuple(2);
> t2.set(0, new Integer(3));
> t2.set(1, new Float(3.0));
> DataBag bag = BagFactory.getInstance().newDefaultBag();
> bag.add(tf.newTuple(new Integer(4)));
> bag.add(tf.newTuple(new String("mary had a little lamb")));
> Map<String, Object> map = new HashMap<String, Object>(2);
> map.put(new String("hello"), new String("world"));
> map.put(new String("goodbye"), new String("all"));
> t1.set(0, t2);
> t1.set(1, bag);
> t1.set(2, map);//when run t.toString(), HashMap.toString() will be invoked
> t1.set(3, new Integer(42));
> t1.set(4, new Long(5000000000L));
> t1.set(5, new Float(3.141592654));
> t1.set(6, new Double(2.99792458e8));
> t1.set(7, new Boolean(true));
> t1.set(8, new DataByteArray("hello"));
> t1.set(9, new String("goodbye"));
> return t1;
> }
> }
> 2.
> [junit] Running org.apache.pig.test.TestLogToPhyCompiler
> [junit] Tests run: 23, Failures: 1, Errors: 0, Time elapsed: 1.16 sec
> Maybe caused by different output of HashMap.keySet() with open source JDK and sun jdk.
> Detail:
> Failure information:
> Testcase: testSplit took 0.226 sec
> FAILED
> Plan not match
> junit.framework.AssertionFailedError: Plan not match
> at org.apache.pig.test.TestLogToPhyCompiler.testSplit(TestLogToPhyCompiler.java:444)
> public void testSplit() throws VisitorException, IOException {
> String query = "split (load 'a') into x if $0 < '7', y if $0 > '7';";
> LogicalPlan plan = buildPlan(query);
> log.info("ff test plan:"+plan);
> PhysicalPlan pp = buildPhysicalPlan(plan);
> log.info("ff test pp:"+pp);
> int MAX_SIZE = 100000;
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
>
> pp.explain(baos);
> baos.write((int)'\n');
> String compiledPlan = baos.toString();
> compiledPlan = compiledPlan.replaceAll("Load(.*)","Load()");
> if(generate){
> FileOutputStream fos = new FileOutputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
> fos.write(baos.toByteArray());
> return;
> }
>
> FileInputStream fis1 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
> FileInputStream fis2 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split2.gld");
> byte[] b1 = new byte[MAX_SIZE];
> byte[] b2 = new byte[MAX_SIZE];
> int len = fis1.read(b1);
> int test = fis2.read(b2);
> //System.out.println("Length of first plan = " + len + " of second = " + test);
> String goldenPlan1 = new String(b1, 0, len);
> String goldenPlan2 = new String(b2, 0, len);
> goldenPlan1 = goldenPlan1.replaceAll("Load(.*)","Load()");
> goldenPlan2 = goldenPlan2.replaceAll("Load(.*)","Load()");
> System.out.println();
> System.out.println(compiledPlan);
> System.out.println("-------------");
> if(compiledPlan.compareTo(goldenPlan1) == 0 || compiledPlan.compareTo(goldenPlan2) == 0) {
> // good
> }
> else {
> System.out.println("Expected plan1=") ;
> System.out.println(goldenPlan1) ;
> System.out.println("Expected plan2=") ;
> System.out.println(goldenPlan1) ;
> System.out.println("Actual plan=") ;
> System.out.println(compiledPlan) ;
> System.out.println("**END**") ;
> fail("Plan not match") ;//line 444
> }
>
> }
> comment:
> variable compiledPlan initialize invoke the following methods:
> pp.explain(baos);// explain(OutputStream out) method
> baos.write((int)'\n');
> String compiledPlan = baos.toString();
> explain(OutputStream out)explain(OutputStream out, boolean verbose)print(OutputStream printer)depthFirstPP()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> baos variable output:
> > open source jdk:
> > x: Filter[tuple] - Test-Plan-Builder-240
> > | |
> > | Less Than[boolean] - Test-Plan-Builder-243
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-241
> > | |
> > | |---Constant(7) - Test-Plan-Builder-242
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> >
> > y: Filter[tuple] - Test-Plan-Builder-244
> > | |
> > | Greater Than[boolean] - Test-Plan-Builder-247
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-245
> > | |
> > | |---Constant(7) - Test-Plan-Builder-246
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> > sun jdk:
> > y: Filter[tuple] - Test-Plan-Builder-240
> > | |
> > | Greater Than[boolean] - Test-Plan-Builder-243
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-241
> > | |
> > | |---Constant(7) - Test-Plan-Builder-242
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> >
> > x: Filter[tuple] - Test-Plan-Builder-244
> > | |
> > | Less Than[boolean] - Test-Plan-Builder-247
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-245
> > | |
> > | |---Constant(7) - Test-Plan-Builder-246
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> 3.
> [junit] Running org.apache.pig.test.TestMRCompiler
> [junit] Tests run: 25, Failures: 1, Errors: 0, Time elapsed: 0.729 sec
> Maybe, caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> Detail:
> Testcase: testSortUDF1 took 0.02 sec
> FAILED
> null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
> | ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> junit.framework.ComparisonFailure: null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
> | ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
> at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
> private void run(PhysicalPlan pp, String expectedFile) throws Exception {
> String compiledPlan, goldenPlan = null;
> int MAX_SIZE = 100000;
> MRCompiler comp = new MRCompiler(pp, pc);
> comp.compile();
> MROperPlan mrp = comp.getMRPlan();
> PlanPrinter ppp = new PlanPrinter(mrp);
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
> ppp.print(baos);//see "comments"
> compiledPlan = baos.toString();//compiledPlan's initialize is based on baos
> if(generate ){
> FileOutputStream fos = new FileOutputStream(expectedFile);
> fos.write(baos.toByteArray());
> return;
> }
> FileInputStream fis = new FileInputStream(expectedFile);
> byte[] b = new byte[MAX_SIZE];
> int len = fis.read(b);
> goldenPlan = new String(b, 0, len);
> if (goldenPlan.charAt(len-1) == '\n')
> goldenPlan = goldenPlan.substring(0, len-1);
> pp.explain(System.out);
> System.out.println();
> System.out.println("<<<" + compiledPlan + ">>>");
> System.out.println("-------------");
> System.out.println("Golden");
> System.out.println("<<<" + goldenPlan + ">>>");
> System.out.println("-------------");
> assertEquals(goldenPlan, compiledPlan);//line 1056
> }
> comment:
> ppp.print(baos) invokes method as following:
> print(OutputStream printer) depthFirstPP()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> 4.
> [junit] Running org.apache.pig.test.TestMergeJoinOuter
> [junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 132.66 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> Testcase: testCompilation took 0.443 sec
> FAILED
> junit.framework.AssertionFailedError:
> at org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116)
> Iterator<MapReduceOper> itr = mrPlan.iterator();// see comments
> MapReduceOper oper = itr.next();
> assertTrue(oper.reducePlan.isEmpty());//line 116
> comments:
> iterator() method:
> public Iterator<E> iterator() {
> return mOps.keySet().iterator();//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> }
> with same mrPlan:
> MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
> Reduce Plan Empty
> | C: Store(hdfs://localhost.localdomain:34390/user/root/out:org.apache.pig.builtin.PigStorage) - scope-38
> | |
> | |---C: New For Each(true,true)[tuple] - scope-37
> | | |
> | | Project[bag][1] - scope-31
> | | |
> | | POBinCond[bag] - scope-36
> | | |
> | | |---Project[bag][2] - scope-32
> | | |
> | | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - scope-34
> | | | |
> | | | |---Project[bag][2] - scope-33
> | | |
> | | |---Constant({(,,)}) - scope-35
> | |
> | |---C: MergeCogroup[tuple] - scope-30
> | |
> | |---A: Load(hdfs://localhost.localdomain:34390/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader) - scope-22
> |
> |---MapReduce(1) - scope-41:
> | Store(hdfs://localhost.localdomain:34390/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage) - scope-48
> | |
> | |---New For Each(true)[bag] - scope-47
> | | |
> | | Project[tuple][1] - scope-46
> | |
> | |---Package[tuple]{tuple} - scope-45
> | Local Rearrange[tuple]{tuple}(false) - scope-44
> | | |
> | | Project[tuple][*] - scope-43
> | |
> variable itr is in different order:
> open source jdk:
> (Name: MapReduce(1) - scope-41(itr.next())
> (Name: MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
> sun jdk:
> (Name: MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) - scope-39: (itr.next())
> (Name: MapReduce(1) - scope-41:
> 5.
> [junit] Running org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor
> [junit] Tests run: 25, Failures: 3, Errors: 0, Time elapsed: 1.081 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1).
> Testcase: testSimplePlan took 0.295 sec
> FAILED
> expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
> junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testSimplePlan(TestNewPlanLogToPhyTranslationVisitor.java:131)
> public void testSimplePlan() throws Exception {
> LogicalPlanTester lpt = new LogicalPlanTester(pc);
> lpt.buildPlan("a = load 'd.txt';");
> lpt.buildPlan("b = filter a by $0==NULL;");
> LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
>
> org.apache.pig.newplan.logical.relational.LogicalPlan newLogicalPlan = migratePlan(plan);
> PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
>
> assertEquals( 3, phyPlan.size() );
> assertEquals( 1, phyPlan.getRoots().size() );
> assertEquals( 1, phyPlan.getLeaves().size() );
>
> PhysicalOperator load = phyPlan.getRoots().get(0);
> assertEquals( POLoad.class, load.getClass() );
> assertTrue( ((POLoad)load).getLFile().getFileName().contains("d.txt") );
>
> // Check for Filter
> PhysicalOperator fil = phyPlan.getSuccessors(load).get(0);
> assertEquals( POFilter.class, fil.getClass() );
> PhysicalPlan filPlan = ((POFilter)fil).getPlan();
> assertEquals( 2, filPlan.getRoots().size() );
> assertEquals( 1, filPlan.getLeaves().size() );
>
> PhysicalOperator eq = filPlan.getLeaves().get(0);
> assertEquals( EqualToExpr.class, eq.getClass() );
>
> PhysicalOperator prj1 = filPlan.getRoots().get(0);
> assertEquals( POProject.class, prj1.getClass() );//line 131
> assertEquals( 0, ((POProject)prj1).getColumn() );
> PhysicalOperator constExp = filPlan.getRoots().get(1);
> assertEquals( ConstantExpression.class, constExp.getClass() );
> assertEquals( null, ((ConstantExpression)constExp).getValue() );
>
> // Check for Store
> PhysicalOperator stor = phyPlan.getSuccessors(fil).get(0);
> assertEquals( POStore.class, stor.getClass() );
> assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty"));
> }
> comment:
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
>
> (2).
> Testcase: testJoinPlan took 0.062 sec
> FAILED
> expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
> junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testJoinPlan(TestNewPlanLogToPhyTranslationVisitor.java:201)
> public void testJoinPlan() throws Exception {
> LogicalPlanTester lpt = new LogicalPlanTester(pc);
> lpt.buildPlan("a = load 'd1.txt' as (id, c);");
> lpt.buildPlan("b = load 'd2.txt'as (id, c);");
> lpt.buildPlan("c = join a by id, b by c;");
> lpt.buildPlan("d = filter c by a::id==NULL AND b::c==NULL;");
> LogicalPlan plan = lpt.buildPlan("store d into 'empty';");
>
> // check basics
> org.apache.pig.newplan.logical.relational.LogicalPlan newPlan = migratePlan(plan);
> PhysicalPlan physicalPlan = translatePlan(newPlan);
> assertEquals(9, physicalPlan.size());
> assertEquals(physicalPlan.getRoots().size(), 2);
>
> // Check Load and LocalRearrange and GlobalRearrange
> PhysicalOperator LoR = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(0)).get(0);
> assertEquals( POLocalRearrange.class, LoR.getClass() );
> POLocalRearrange Lor = (POLocalRearrange) LoR;
> PhysicalOperator prj3 = Lor.getPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj3.getClass() );
> assertEquals(0, ((POProject)prj3).getColumn() );
> PhysicalOperator inp1 = Lor.getInputs().get(0);
> assertEquals( POLoad.class, inp1.getClass() );
> assertTrue( ((POLoad)inp1).getLFile().getFileName().contains("d1.txt") );
>
> PhysicalOperator LoR1 = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(1)).get(0);
> assertEquals( POLocalRearrange.class, LoR1.getClass() );
> POLocalRearrange Lor1 = (POLocalRearrange) LoR1;
> PhysicalOperator prj4 = Lor1.getPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj4.getClass() );
> assertEquals(1, ((POProject)prj4).getColumn() );
> PhysicalOperator inp2 = Lor1.getInputs().get(0);
> assertEquals( POLoad.class, inp2.getClass() );
> assertTrue( ((POLoad)inp2).getLFile().getFileName().contains("d2.txt") );
>
> PhysicalOperator GoR = (PhysicalOperator)physicalPlan.getSuccessors(LoR).get(0);
> assertEquals( POGlobalRearrange.class, GoR.getClass() );
>
> PhysicalOperator Pack = (PhysicalOperator)physicalPlan.getSuccessors(GoR).get(0);
> assertEquals( POPackage.class, Pack.getClass() );
> // Check for ForEach
> PhysicalOperator ForE = (PhysicalOperator)physicalPlan.getSuccessors(Pack).get(0);
> assertEquals( POForEach.class, ForE.getClass() );
> PhysicalOperator prj5 = ((POForEach)ForE).getInputPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj5.getClass() );
> assertEquals( 1, ((POProject)prj5).getColumn() );
> PhysicalOperator prj6 = ((POForEach)ForE).getInputPlans().get(1).getLeaves().get(0);
> assertEquals( POProject.class, prj6.getClass() );
> assertEquals( 2, ((POProject)prj6).getColumn() );
>
> // Filter Operator
> PhysicalOperator fil = (PhysicalOperator)physicalPlan.getSuccessors(ForE).get(0);
> assertEquals( POFilter.class, fil.getClass() );
>
> PhysicalPlan filPlan = ((POFilter)fil).getPlan();
> List<PhysicalOperator> filRoots = filPlan.getRoots();
>
> assertEquals( ConstantExpression.class, filRoots.get(1).getClass() );//line 201
> ConstantExpression ce1 = (ConstantExpression) filRoots.get(1);
> assertEquals( null, ce1.getValue() );
> assertEquals( ConstantExpression.class, filRoots.get(3).getClass() );
> ConstantExpression ce2 = (ConstantExpression) filRoots.get(3);
> assertEquals( null, ce2.getValue() );
> assertEquals( POProject.class, filRoots.get(0).getClass() );
> POProject prj1 = (POProject) filRoots.get(0);
> assertEquals( 3, prj1.getColumn() );
> assertEquals( POProject.class, filRoots.get(2).getClass() );
> POProject prj2 = (POProject) filRoots.get(2);
> assertEquals( 0, prj2.getColumn() );
> // Check Store Operator
> PhysicalOperator stor = (PhysicalOperator)physicalPlan.getSuccessors(fil).get(0);
> assertEquals( POStore.class, stor.getClass() );
> assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty") );
> }
> comment:
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> (3).
> Testcase: testMultiStore took 0.083 sec
> FAILED
> expected:<1> but was:<0>
> junit.framework.AssertionFailedError: expected:<1> but was:<0>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testMultiStore(TestNewPlanLogToPhyTranslationVisitor.java:255)
> PhysicalOperator prj2 = Lor1.getPlans().get(0).getLeaves().get(0);
> assertEquals(1, ((POProject)prj2).getColumn() );//line 255
> comment:
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> 6.
> [junit] Running org.apache.pig.test.TestPruneColumn
> [junit] Tests run: 67, Failures: 4, Errors: 0, Time elapsed: 528.047 sec
> Maybe caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1).
> Testcase: testMapKey2 took 6.291 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKey2(TestPruneColumn.java:1213)
> public void testMapKey2() throws Exception{
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
> pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");// see comment3
> pigServer.registerQuery("C = foreach B generate $0#'key2', $1;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");//see comment1
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("2"));
> assertTrue(t.get(1).toString().equals("1"));
>
> assertTrue(iter.hasNext());
> t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("4"));
> assertTrue(t.get(1).toString().equals("2"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0",
> "Map key required for A: $1->[key2, key1]"}));// line 1213 see comment2
> }
> comments1:
> pigServer.openIterator("C") invokes the following method to save pigServer information to filename:
> store(String id, String filename, String func)
> comment2:
> public boolean checkLogFileMessage(String[] messages)
> {
> BufferedReader reader = null;
>
> try {
> reader = new BufferedReader(new FileReader(logFile));//logFile=filename
> List<String> logMessages=new ArrayList<String>();
> String line;
> while ((line=reader.readLine())!=null)
> {
> logMessages.add(line);
> }
>
> // Check if all messages appear in the log
> for (int i=0;i<messages.length;i++)
> {
> boolean found = false;
> for (int j=0;j<logMessages.size();j++)
> if (logMessages.get(j).contains(messages[i])) {
> found = true;
> break;
> }
> if (!found)
> return false;
> }
>
> // Check no other log besides messages
> for (int i=0;i<logMessages.size();i++) {
> boolean found = false;
> for (int j=0;j<messages.length;j++) {
> if (logMessages.get(i).contains(messages[j])) {
> found = true;
> break;
> }
> }
> if (!found) {
> if (logMessages.get(i).contains("Columns pruned for")||
> logMessages.get(i).contains("Map key required for")) {
> return false;
> }
> }
> }
> return true;
> }
> catch (IOException e) {
> return false;
> }
> }
> comment3:
> the content in filename is different, because pigServer(invoke HashMap.keySet()) is different.
> pigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> (2)
> Testcase: testMapKey3 took 6.319 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKey3(TestPruneColumn.java:1229)
> public void testMapKey3() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
> pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");
> pigServer.registerQuery("C = group B all;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("all"));
> assertTrue(t.get(1).toString().equals("{([key2#2,key1#1],1),([key2#4,key1#2],2)}"));
> //line 1229
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0"}));
> }
> comment:
> variable "t" initialize process:
> titerpigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> different t.get(1).toString():
> OPEN SOURCE JDK:
> {([key1#1,key2#2],1),([key1#2,key2#4],2)}
> SUN JDK:
> {([key2#2,key1#1],1),([key2#4,key1#2],2)}
> (3).
> Testcase: testMapKeyInSplit1 took 6.3 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKeyInSplit1(TestPruneColumn.java:1303)
> public void testMapKeyInSplit1() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as (m:map[]);");
> pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
> pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
> pigServer.registerQuery("D = join B by key1, C by key2;");
>
> Iterator<Tuple> iter = pigServer.openIterator("D");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("2"));
> assertTrue(t.get(1).toString().equals("2"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Map key required for A: $0->[key2, key1]"}));//line 1303
> }
> comment: same with (1).
> (4).
> Testcase: testSharedSchemaObject took 6.327 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testSharedSchemaObject(TestPruneColumn.java:1626)
> public void testSharedSchemaObject() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile10.toString(), pigServer.getPigContext()) + "' AS (a0, a1:map[], a2);");
> pigServer.registerQuery("B = foreach A generate a1;");
> pigServer.registerQuery("C = limit B 10;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.toString().equals("([2#1,1#1])"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0, $2"}));// line 1626
> }
> comment: same with (2).
> 7.
> [junit] Running org.apache.pig.test.TestUnionOnSchema
> [junit] Tests run: 21, Failures: 1, Errors: 0, Time elapsed: 196.841 sec
> Testcase: testUnionOnSchemaScopedColumnNameNeg took 0.008 sec
> FAILED
> Expected exception message matching 'Found more than one match: l1::i, l2::i' but got 'Error during parsing. Found more than one match: l2::i, l1::i'
> at org.apache.pig.test.TestUnionOnSchema.checkSchemaEx(TestUnionOnSchema.java:604)
> at org.apache.pig.test.TestUnionOnSchema.testUnionOnSchemaScopedColumnNameNeg(TestUnionOnSchema.java:370)
> 8..
> [junit] Running org.apache.pig.test.TestPushDownForeachFlatten
> [junit] Tests run: 37, Failures: 0, Errors: 8, Time elapsed: 1.455 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1)
> Testcase: testForeachUnion took 0.039 sec
> Caused an ERROR
> Expected LOForEach, got LOUnion
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOUnion
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachUnion(TestPushDownForeachFlatten.java:275)
> public void testForeachUnion() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> LogicalPlan lp = planTester.buildPlan("D = union B, C;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
>
> LOLoad load = (LOLoad) lp.getRoots().get(0);// see comment
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 275
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment:
> LOLoad load = (LOLoad) lp.getRoots().get(0);//
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> (2)
> Testcase: testForeachCogroup took 0.038 sec
> Caused an ERROR
> Expected LOForEach, got LOCogroup
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCogroup
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCogroup(TestPushDownForeachFlatten.java:295)
> public void testForeachCogroup() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> LogicalPlan lp = planTester.buildPlan("D = cogroup B by $0, C by $0;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
>
> LOLoad load = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 295
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (3)
> Testcase: testForeachCross took 0.035 sec
> Caused an ERROR
> Expected LOForEach, got LOCross
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCross(TestPushDownForeachFlatten.java:427)
> public void testForeachCross() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> planTester.buildPlan("D = cross B, C;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOCross cross = (LOCross)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(cross).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));//line 427
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(cross).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
>
> }
> comment: same with (1)
> (4)
> Testcase: testForeachFlattenAddedColumnCross took 0.034 sec
> Caused an ERROR
> Expected LOForEach, got LOCross
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnCross(TestPushDownForeachFlatten.java:545)
> public void testForeachFlattenAddedColumnCross() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = cross B, C;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 545
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (5)
> Testcase: testForeachFRJoin took 0.027 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFRJoin(TestPushDownForeachFlatten.java:619)
> public void testForeachFRJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOJoin frjoin = (LOJoin)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(frjoin).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 619
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(frjoin).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
> }
> comment: same with (1)
> (6)
> Testcase: testForeachFlattenAddedColumnFRJoin took 0.026 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnFRJoin(TestPushDownForeachFlatten.java:738)
> public void testForeachFlattenAddedColumnFRJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 738
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (7)
> Testcase: testForeachInnerJoin took 0.026 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachInnerJoin(TestPushDownForeachFlatten.java:812)
> public void testForeachInnerJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOJoin join = (LOJoin)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(join).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 812
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(join).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
> }
> comment: same with (1)
> (8)
> Testcase: testForeachFlattenAddedColumnInnerJoin took 0.021 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnInnerJoin(TestPushDownForeachFlatten.java:931)
> public void testForeachFlattenAddedColumnInnerJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 931
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> 9.
> [junit] Running org.apache.pig.test.TestTypeCheckingValidator
> [junit] Tests run: 120, Failures: 0, Errors: 1, Time elapsed: 15.047 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in OPEN SOURCE jdk is also correct.
> Detail:
> Testcase: testMapLookupLineage took 0.012 sec
> Caused an ERROR
> org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
> java.lang.ClassCastException: org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
> at org.apache.pig.test.TestTypeCheckingValidator.testMapLookupLineage(TestTypeCheckingValidator.java:5397)
> public void testMapLookupLineage() throws Throwable {
> planTester.buildPlan("a = load 'a' using BinStorage() as (field1, field2: float, field3: chararray );") ;
> planTester.buildPlan("b = foreach a generate field1#'key1' as map1;") ;
> LogicalPlan plan = planTester.buildPlan("c = foreach b generate map1#'key2' + 1 ;") ;
> // validate
> CompilationMessageCollector collector = new CompilationMessageCollector() ;
> TypeCheckingValidator typeValidator = new TypeCheckingValidator() ;
> typeValidator.validate(plan, collector) ;
> printMessageCollector(collector) ;
> printTypeGraph(plan) ;
> planTester.printPlan(plan, TypeCheckingTestUtil.getCurrentMethodName());
> if (collector.hasError()) {
> throw new AssertionError("Expect no error") ;
> }
> LOForEach foreach = (LOForEach)plan.getLeaves().get(0);// see comement1
> LogicalPlan foreachPlan = foreach.getForEachPlans().get(0);
> LogicalOperator exOp = foreachPlan.getRoots().get(0);//see comment2
> // the root would be the project and there would be cast
> // to map between the project and LOMapLookup
> LOCast cast1 = (LOCast)foreachPlan.getSuccessors(exOp).get(0);//line 5397
> assertTrue(cast1.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
> LOMapLookup map = (LOMapLookup)foreachPlan.getSuccessors(cast1).get(0);
> LOCast cast = (LOCast)foreachPlan.getSuccessors(map).get(0);
> assertTrue(cast.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
> }
> comment1:
> foreachPlan initialize process, this will cause the different foreachPlan.
> foreachPlanforeachplan.getLeaves
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> comment2:
> exOpforeachPlan.getRoots(), this is the root cause
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> SUN JDK output:
> foreachPlan:
> Add scope-12 FieldSchema: int Type: int
> |
> |---Const scope-15( 1 ) FieldSchema: int Type: int
> |
> |---Cast scope-19 FieldSchema: int Type: int
> |
> |---MapLookup scope-14 FieldSchema: bytearray Type: bytearray
> |
> |---Cast scope-18 FieldSchema: map Type: map
> |
> |---Project scope-13 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
> Input: b: ForEach scope-6
> getRoots method process:
> keySet() method:
> Cast scope-19
> Cast scope-18
> Add scope-12
> Project scope-13 Projections: [0] Overloaded: false(add to mRoots)
> MapLookup scope-14
> Const scope-15( 1 )(add to mRoots)
> foreachPlan.getRoots():(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
> (Name: Const scope-15( 1 ) Operator Key: scope-15)
> exOp:(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
> OPEN SOURCE JDK output:
> foreachPlan:
> Add scope-13 FieldSchema: int Type: int
> |
> |---Const scope-12( 1 ) FieldSchema: int Type: int
> |
> |---Cast scope-19 FieldSchema: int Type: int
> |
> |---MapLookup scope-15 FieldSchema: bytearray Type: bytearray
> |
> |---Cast scope-18 FieldSchema: map Type: map
> |
> |---Project scope-14 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
> Input: b: ForEach scope-6
> getRoots method process:
> keySet() method:
> Cast scope-18
> Cast scope-19
> Const scope-12( 1 )(add to mRoots)
> Add scope-13
> Project scope-14 Projections: [0] Overloaded: false(add to mRoots)
> MapLookup scope-15
> foreachPlan.getRoots():(Name: Const scope-12( 1 ) Operator Key: scope-12)
> (Name: Project scope-14 Projections: [0] Overloaded: false Operator Key: scope-14)//output in different order compared with SUN JDK output
> exOp:(Name: Const scope-12( 1 ) Operator Key: scope-12)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2281) pig-0.8.1: some unit test case failed
with open source JDK
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104749#comment-13104749 ]
Daniel Dai commented on PIG-2281:
---------------------------------
Have you tested against 0.9? We use ArrayList instead of HashMap in 0.9.
> pig-0.8.1: some unit test case failed with open source JDK
> ----------------------------------------------------------
>
> Key: PIG-2281
> URL: https://issues.apache.org/jira/browse/PIG-2281
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: open source jdk
> ant: 1.8.2
> Reporter: fang fang chen
> Fix For: 0.8.1
>
>
> Based on my analysis, I think this is because HashMap does not guarantee ordering.
> 1.
> [junit] Running org.apache.pig.test.TestDataModel
> [junit] Tests run: 22, Failures: 1, Errors: 0, Time elapsed: 0.382 sec
> Caused by different output of HashMap.toString() with open source JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in open source jdk is also correct.
> Detail:
> Testcase: testTupleToString took 0.002 sec
> FAILED
> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
> public void testTupleToString() throws Exception {
> Tuple t = giveMeOneOfEach();
> assertEquals("toString", "((3,3.0),{(4),(mary had a little lamb)},[hello#world,goodbye#all],42,5000000000,3.1415927,2.99792458E8,true,hello,goodbye,)", t.toString());//line 269
> }
> comment:
> private Tuple giveMeOneOfEach() throws Exception {
> TupleFactory tf = TupleFactory.getInstance();
> Tuple t1 = tf.newTuple(11);
> Tuple t2 = tf.newTuple(2);
> t2.set(0, new Integer(3));
> t2.set(1, new Float(3.0));
> DataBag bag = BagFactory.getInstance().newDefaultBag();
> bag.add(tf.newTuple(new Integer(4)));
> bag.add(tf.newTuple(new String("mary had a little lamb")));
> Map<String, Object> map = new HashMap<String, Object>(2);
> map.put(new String("hello"), new String("world"));
> map.put(new String("goodbye"), new String("all"));
> t1.set(0, t2);
> t1.set(1, bag);
> t1.set(2, map);//when run t.toString(), HashMap.toString() will be invoked
> t1.set(3, new Integer(42));
> t1.set(4, new Long(5000000000L));
> t1.set(5, new Float(3.141592654));
> t1.set(6, new Double(2.99792458e8));
> t1.set(7, new Boolean(true));
> t1.set(8, new DataByteArray("hello"));
> t1.set(9, new String("goodbye"));
> return t1;
> }
> }
> 2.
> [junit] Running org.apache.pig.test.TestLogToPhyCompiler
> [junit] Tests run: 23, Failures: 1, Errors: 0, Time elapsed: 1.16 sec
> Maybe caused by different output of HashMap.keySet() with open source JDK and sun jdk.
> Detail:
> Failure information:
> Testcase: testSplit took 0.226 sec
> FAILED
> Plan not match
> junit.framework.AssertionFailedError: Plan not match
> at org.apache.pig.test.TestLogToPhyCompiler.testSplit(TestLogToPhyCompiler.java:444)
> public void testSplit() throws VisitorException, IOException {
> String query = "split (load 'a') into x if $0 < '7', y if $0 > '7';";
> LogicalPlan plan = buildPlan(query);
> log.info("ff test plan:"+plan);
> PhysicalPlan pp = buildPhysicalPlan(plan);
> log.info("ff test pp:"+pp);
> int MAX_SIZE = 100000;
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
>
> pp.explain(baos);
> baos.write((int)'\n');
> String compiledPlan = baos.toString();
> compiledPlan = compiledPlan.replaceAll("Load(.*)","Load()");
> if(generate){
> FileOutputStream fos = new FileOutputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
> fos.write(baos.toByteArray());
> return;
> }
>
> FileInputStream fis1 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
> FileInputStream fis2 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split2.gld");
> byte[] b1 = new byte[MAX_SIZE];
> byte[] b2 = new byte[MAX_SIZE];
> int len = fis1.read(b1);
> int test = fis2.read(b2);
> //System.out.println("Length of first plan = " + len + " of second = " + test);
> String goldenPlan1 = new String(b1, 0, len);
> String goldenPlan2 = new String(b2, 0, len);
> goldenPlan1 = goldenPlan1.replaceAll("Load(.*)","Load()");
> goldenPlan2 = goldenPlan2.replaceAll("Load(.*)","Load()");
> System.out.println();
> System.out.println(compiledPlan);
> System.out.println("-------------");
> if(compiledPlan.compareTo(goldenPlan1) == 0 || compiledPlan.compareTo(goldenPlan2) == 0) {
> // good
> }
> else {
> System.out.println("Expected plan1=") ;
> System.out.println(goldenPlan1) ;
> System.out.println("Expected plan2=") ;
> System.out.println(goldenPlan1) ;
> System.out.println("Actual plan=") ;
> System.out.println(compiledPlan) ;
> System.out.println("**END**") ;
> fail("Plan not match") ;//line 444
> }
>
> }
> comment:
> variable compiledPlan initialize invoke the following methods:
> pp.explain(baos);// explain(OutputStream out) method
> baos.write((int)'\n');
> String compiledPlan = baos.toString();
> explain(OutputStream out)explain(OutputStream out, boolean verbose)print(OutputStream printer)depthFirstPP()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> baos variable output:
> > open source jdk:
> > x: Filter[tuple] - Test-Plan-Builder-240
> > | |
> > | Less Than[boolean] - Test-Plan-Builder-243
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-241
> > | |
> > | |---Constant(7) - Test-Plan-Builder-242
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> >
> > y: Filter[tuple] - Test-Plan-Builder-244
> > | |
> > | Greater Than[boolean] - Test-Plan-Builder-247
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-245
> > | |
> > | |---Constant(7) - Test-Plan-Builder-246
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> > sun jdk:
> > y: Filter[tuple] - Test-Plan-Builder-240
> > | |
> > | Greater Than[boolean] - Test-Plan-Builder-243
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-241
> > | |
> > | |---Constant(7) - Test-Plan-Builder-242
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> >
> > x: Filter[tuple] - Test-Plan-Builder-244
> > | |
> > | Less Than[boolean] - Test-Plan-Builder-247
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-245
> > | |
> > | |---Constant(7) - Test-Plan-Builder-246
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> 3.
> [junit] Running org.apache.pig.test.TestMRCompiler
> [junit] Tests run: 25, Failures: 1, Errors: 0, Time elapsed: 0.729 sec
> Maybe, caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> Detail:
> Testcase: testSortUDF1 took 0.02 sec
> FAILED
> null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
> | ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> junit.framework.ComparisonFailure: null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
> | ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
> at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
> private void run(PhysicalPlan pp, String expectedFile) throws Exception {
> String compiledPlan, goldenPlan = null;
> int MAX_SIZE = 100000;
> MRCompiler comp = new MRCompiler(pp, pc);
> comp.compile();
> MROperPlan mrp = comp.getMRPlan();
> PlanPrinter ppp = new PlanPrinter(mrp);
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
> ppp.print(baos);//see "comments"
> compiledPlan = baos.toString();//compiledPlan's initialize is based on baos
> if(generate ){
> FileOutputStream fos = new FileOutputStream(expectedFile);
> fos.write(baos.toByteArray());
> return;
> }
> FileInputStream fis = new FileInputStream(expectedFile);
> byte[] b = new byte[MAX_SIZE];
> int len = fis.read(b);
> goldenPlan = new String(b, 0, len);
> if (goldenPlan.charAt(len-1) == '\n')
> goldenPlan = goldenPlan.substring(0, len-1);
> pp.explain(System.out);
> System.out.println();
> System.out.println("<<<" + compiledPlan + ">>>");
> System.out.println("-------------");
> System.out.println("Golden");
> System.out.println("<<<" + goldenPlan + ">>>");
> System.out.println("-------------");
> assertEquals(goldenPlan, compiledPlan);//line 1056
> }
> comment:
> ppp.print(baos) invokes method as following:
> print(OutputStream printer) depthFirstPP()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> 4.
> [junit] Running org.apache.pig.test.TestMergeJoinOuter
> [junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 132.66 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> Testcase: testCompilation took 0.443 sec
> FAILED
> junit.framework.AssertionFailedError:
> at org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116)
> Iterator<MapReduceOper> itr = mrPlan.iterator();// see comments
> MapReduceOper oper = itr.next();
> assertTrue(oper.reducePlan.isEmpty());//line 116
> comments:
> iterator() method:
> public Iterator<E> iterator() {
> return mOps.keySet().iterator();//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> }
> with same mrPlan:
> MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
> Reduce Plan Empty
> | C: Store(hdfs://localhost.localdomain:34390/user/root/out:org.apache.pig.builtin.PigStorage) - scope-38
> | |
> | |---C: New For Each(true,true)[tuple] - scope-37
> | | |
> | | Project[bag][1] - scope-31
> | | |
> | | POBinCond[bag] - scope-36
> | | |
> | | |---Project[bag][2] - scope-32
> | | |
> | | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - scope-34
> | | | |
> | | | |---Project[bag][2] - scope-33
> | | |
> | | |---Constant({(,,)}) - scope-35
> | |
> | |---C: MergeCogroup[tuple] - scope-30
> | |
> | |---A: Load(hdfs://localhost.localdomain:34390/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader) - scope-22
> |
> |---MapReduce(1) - scope-41:
> | Store(hdfs://localhost.localdomain:34390/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage) - scope-48
> | |
> | |---New For Each(true)[bag] - scope-47
> | | |
> | | Project[tuple][1] - scope-46
> | |
> | |---Package[tuple]{tuple} - scope-45
> | Local Rearrange[tuple]{tuple}(false) - scope-44
> | | |
> | | Project[tuple][*] - scope-43
> | |
> variable itr is in different order:
> open source jdk:
> (Name: MapReduce(1) - scope-41(itr.next())
> (Name: MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
> sun jdk:
> (Name: MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) - scope-39: (itr.next())
> (Name: MapReduce(1) - scope-41:
> 5.
> [junit] Running org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor
> [junit] Tests run: 25, Failures: 3, Errors: 0, Time elapsed: 1.081 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1).
> Testcase: testSimplePlan took 0.295 sec
> FAILED
> expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
> junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testSimplePlan(TestNewPlanLogToPhyTranslationVisitor.java:131)
> public void testSimplePlan() throws Exception {
> LogicalPlanTester lpt = new LogicalPlanTester(pc);
> lpt.buildPlan("a = load 'd.txt';");
> lpt.buildPlan("b = filter a by $0==NULL;");
> LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
>
> org.apache.pig.newplan.logical.relational.LogicalPlan newLogicalPlan = migratePlan(plan);
> PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
>
> assertEquals( 3, phyPlan.size() );
> assertEquals( 1, phyPlan.getRoots().size() );
> assertEquals( 1, phyPlan.getLeaves().size() );
>
> PhysicalOperator load = phyPlan.getRoots().get(0);
> assertEquals( POLoad.class, load.getClass() );
> assertTrue( ((POLoad)load).getLFile().getFileName().contains("d.txt") );
>
> // Check for Filter
> PhysicalOperator fil = phyPlan.getSuccessors(load).get(0);
> assertEquals( POFilter.class, fil.getClass() );
> PhysicalPlan filPlan = ((POFilter)fil).getPlan();
> assertEquals( 2, filPlan.getRoots().size() );
> assertEquals( 1, filPlan.getLeaves().size() );
>
> PhysicalOperator eq = filPlan.getLeaves().get(0);
> assertEquals( EqualToExpr.class, eq.getClass() );
>
> PhysicalOperator prj1 = filPlan.getRoots().get(0);
> assertEquals( POProject.class, prj1.getClass() );//line 131
> assertEquals( 0, ((POProject)prj1).getColumn() );
> PhysicalOperator constExp = filPlan.getRoots().get(1);
> assertEquals( ConstantExpression.class, constExp.getClass() );
> assertEquals( null, ((ConstantExpression)constExp).getValue() );
>
> // Check for Store
> PhysicalOperator stor = phyPlan.getSuccessors(fil).get(0);
> assertEquals( POStore.class, stor.getClass() );
> assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty"));
> }
> comment:
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
>
> (2).
> Testcase: testJoinPlan took 0.062 sec
> FAILED
> expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
> junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testJoinPlan(TestNewPlanLogToPhyTranslationVisitor.java:201)
> public void testJoinPlan() throws Exception {
> LogicalPlanTester lpt = new LogicalPlanTester(pc);
> lpt.buildPlan("a = load 'd1.txt' as (id, c);");
> lpt.buildPlan("b = load 'd2.txt'as (id, c);");
> lpt.buildPlan("c = join a by id, b by c;");
> lpt.buildPlan("d = filter c by a::id==NULL AND b::c==NULL;");
> LogicalPlan plan = lpt.buildPlan("store d into 'empty';");
>
> // check basics
> org.apache.pig.newplan.logical.relational.LogicalPlan newPlan = migratePlan(plan);
> PhysicalPlan physicalPlan = translatePlan(newPlan);
> assertEquals(9, physicalPlan.size());
> assertEquals(physicalPlan.getRoots().size(), 2);
>
> // Check Load and LocalRearrange and GlobalRearrange
> PhysicalOperator LoR = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(0)).get(0);
> assertEquals( POLocalRearrange.class, LoR.getClass() );
> POLocalRearrange Lor = (POLocalRearrange) LoR;
> PhysicalOperator prj3 = Lor.getPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj3.getClass() );
> assertEquals(0, ((POProject)prj3).getColumn() );
> PhysicalOperator inp1 = Lor.getInputs().get(0);
> assertEquals( POLoad.class, inp1.getClass() );
> assertTrue( ((POLoad)inp1).getLFile().getFileName().contains("d1.txt") );
>
> PhysicalOperator LoR1 = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(1)).get(0);
> assertEquals( POLocalRearrange.class, LoR1.getClass() );
> POLocalRearrange Lor1 = (POLocalRearrange) LoR1;
> PhysicalOperator prj4 = Lor1.getPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj4.getClass() );
> assertEquals(1, ((POProject)prj4).getColumn() );
> PhysicalOperator inp2 = Lor1.getInputs().get(0);
> assertEquals( POLoad.class, inp2.getClass() );
> assertTrue( ((POLoad)inp2).getLFile().getFileName().contains("d2.txt") );
>
> PhysicalOperator GoR = (PhysicalOperator)physicalPlan.getSuccessors(LoR).get(0);
> assertEquals( POGlobalRearrange.class, GoR.getClass() );
>
> PhysicalOperator Pack = (PhysicalOperator)physicalPlan.getSuccessors(GoR).get(0);
> assertEquals( POPackage.class, Pack.getClass() );
> // Check for ForEach
> PhysicalOperator ForE = (PhysicalOperator)physicalPlan.getSuccessors(Pack).get(0);
> assertEquals( POForEach.class, ForE.getClass() );
> PhysicalOperator prj5 = ((POForEach)ForE).getInputPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj5.getClass() );
> assertEquals( 1, ((POProject)prj5).getColumn() );
> PhysicalOperator prj6 = ((POForEach)ForE).getInputPlans().get(1).getLeaves().get(0);
> assertEquals( POProject.class, prj6.getClass() );
> assertEquals( 2, ((POProject)prj6).getColumn() );
>
> // Filter Operator
> PhysicalOperator fil = (PhysicalOperator)physicalPlan.getSuccessors(ForE).get(0);
> assertEquals( POFilter.class, fil.getClass() );
>
> PhysicalPlan filPlan = ((POFilter)fil).getPlan();
> List<PhysicalOperator> filRoots = filPlan.getRoots();
>
> assertEquals( ConstantExpression.class, filRoots.get(1).getClass() );//line 201
> ConstantExpression ce1 = (ConstantExpression) filRoots.get(1);
> assertEquals( null, ce1.getValue() );
> assertEquals( ConstantExpression.class, filRoots.get(3).getClass() );
> ConstantExpression ce2 = (ConstantExpression) filRoots.get(3);
> assertEquals( null, ce2.getValue() );
> assertEquals( POProject.class, filRoots.get(0).getClass() );
> POProject prj1 = (POProject) filRoots.get(0);
> assertEquals( 3, prj1.getColumn() );
> assertEquals( POProject.class, filRoots.get(2).getClass() );
> POProject prj2 = (POProject) filRoots.get(2);
> assertEquals( 0, prj2.getColumn() );
> // Check Store Operator
> PhysicalOperator stor = (PhysicalOperator)physicalPlan.getSuccessors(fil).get(0);
> assertEquals( POStore.class, stor.getClass() );
> assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty") );
> }
> comment:
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> (3).
> Testcase: testMultiStore took 0.083 sec
> FAILED
> expected:<1> but was:<0>
> junit.framework.AssertionFailedError: expected:<1> but was:<0>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testMultiStore(TestNewPlanLogToPhyTranslationVisitor.java:255)
> PhysicalOperator prj2 = Lor1.getPlans().get(0).getLeaves().get(0);
> assertEquals(1, ((POProject)prj2).getColumn() );//line 255
> comment:
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> 6.
> [junit] Running org.apache.pig.test.TestPruneColumn
> [junit] Tests run: 67, Failures: 4, Errors: 0, Time elapsed: 528.047 sec
> Maybe caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1).
> Testcase: testMapKey2 took 6.291 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKey2(TestPruneColumn.java:1213)
> public void testMapKey2() throws Exception{
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
> pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");// see comment3
> pigServer.registerQuery("C = foreach B generate $0#'key2', $1;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");//see comment1
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("2"));
> assertTrue(t.get(1).toString().equals("1"));
>
> assertTrue(iter.hasNext());
> t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("4"));
> assertTrue(t.get(1).toString().equals("2"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0",
> "Map key required for A: $1->[key2, key1]"}));// line 1213 see comment2
> }
> comments1:
> pigServer.openIterator("C") invokes the following method to save pigServer information to filename:
> store(String id, String filename, String func)
> comment2:
> public boolean checkLogFileMessage(String[] messages)
> {
> BufferedReader reader = null;
>
> try {
> reader = new BufferedReader(new FileReader(logFile));//logFile=filename
> List<String> logMessages=new ArrayList<String>();
> String line;
> while ((line=reader.readLine())!=null)
> {
> logMessages.add(line);
> }
>
> // Check if all messages appear in the log
> for (int i=0;i<messages.length;i++)
> {
> boolean found = false;
> for (int j=0;j<logMessages.size();j++)
> if (logMessages.get(j).contains(messages[i])) {
> found = true;
> break;
> }
> if (!found)
> return false;
> }
>
> // Check no other log besides messages
> for (int i=0;i<logMessages.size();i++) {
> boolean found = false;
> for (int j=0;j<messages.length;j++) {
> if (logMessages.get(i).contains(messages[j])) {
> found = true;
> break;
> }
> }
> if (!found) {
> if (logMessages.get(i).contains("Columns pruned for")||
> logMessages.get(i).contains("Map key required for")) {
> return false;
> }
> }
> }
> return true;
> }
> catch (IOException e) {
> return false;
> }
> }
> comment3:
> the content in filename is different, because pigServer(invoke HashMap.keySet()) is different.
> pigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> (2)
> Testcase: testMapKey3 took 6.319 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKey3(TestPruneColumn.java:1229)
> public void testMapKey3() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
> pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");
> pigServer.registerQuery("C = group B all;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("all"));
> assertTrue(t.get(1).toString().equals("{([key2#2,key1#1],1),([key2#4,key1#2],2)}"));
> //line 1229
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0"}));
> }
> comment:
> variable "t" initialize process:
> titerpigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> different t.get(1).toString():
> OPEN SOURCE JDK:
> {([key1#1,key2#2],1),([key1#2,key2#4],2)}
> SUN JDK:
> {([key2#2,key1#1],1),([key2#4,key1#2],2)}
> (3).
> Testcase: testMapKeyInSplit1 took 6.3 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKeyInSplit1(TestPruneColumn.java:1303)
> public void testMapKeyInSplit1() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as (m:map[]);");
> pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
> pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
> pigServer.registerQuery("D = join B by key1, C by key2;");
>
> Iterator<Tuple> iter = pigServer.openIterator("D");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("2"));
> assertTrue(t.get(1).toString().equals("2"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Map key required for A: $0->[key2, key1]"}));//line 1303
> }
> comment: same with (1).
> (4).
> Testcase: testSharedSchemaObject took 6.327 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testSharedSchemaObject(TestPruneColumn.java:1626)
> public void testSharedSchemaObject() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile10.toString(), pigServer.getPigContext()) + "' AS (a0, a1:map[], a2);");
> pigServer.registerQuery("B = foreach A generate a1;");
> pigServer.registerQuery("C = limit B 10;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.toString().equals("([2#1,1#1])"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0, $2"}));// line 1626
> }
> comment: same with (2).
> 7.
> [junit] Running org.apache.pig.test.TestUnionOnSchema
> [junit] Tests run: 21, Failures: 1, Errors: 0, Time elapsed: 196.841 sec
> Testcase: testUnionOnSchemaScopedColumnNameNeg took 0.008 sec
> FAILED
> Expected exception message matching 'Found more than one match: l1::i, l2::i' but got 'Error during parsing. Found more than one match: l2::i, l1::i'
> at org.apache.pig.test.TestUnionOnSchema.checkSchemaEx(TestUnionOnSchema.java:604)
> at org.apache.pig.test.TestUnionOnSchema.testUnionOnSchemaScopedColumnNameNeg(TestUnionOnSchema.java:370)
> 8..
> [junit] Running org.apache.pig.test.TestPushDownForeachFlatten
> [junit] Tests run: 37, Failures: 0, Errors: 8, Time elapsed: 1.455 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1)
> Testcase: testForeachUnion took 0.039 sec
> Caused an ERROR
> Expected LOForEach, got LOUnion
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOUnion
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachUnion(TestPushDownForeachFlatten.java:275)
> public void testForeachUnion() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> LogicalPlan lp = planTester.buildPlan("D = union B, C;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
>
> LOLoad load = (LOLoad) lp.getRoots().get(0);// see comment
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 275
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment:
> LOLoad load = (LOLoad) lp.getRoots().get(0);//
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> (2)
> Testcase: testForeachCogroup took 0.038 sec
> Caused an ERROR
> Expected LOForEach, got LOCogroup
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCogroup
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCogroup(TestPushDownForeachFlatten.java:295)
> public void testForeachCogroup() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> LogicalPlan lp = planTester.buildPlan("D = cogroup B by $0, C by $0;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
>
> LOLoad load = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 295
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (3)
> Testcase: testForeachCross took 0.035 sec
> Caused an ERROR
> Expected LOForEach, got LOCross
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCross(TestPushDownForeachFlatten.java:427)
> public void testForeachCross() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> planTester.buildPlan("D = cross B, C;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOCross cross = (LOCross)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(cross).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));//line 427
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(cross).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
>
> }
> comment: same with (1)
> (4)
> Testcase: testForeachFlattenAddedColumnCross took 0.034 sec
> Caused an ERROR
> Expected LOForEach, got LOCross
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnCross(TestPushDownForeachFlatten.java:545)
> public void testForeachFlattenAddedColumnCross() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = cross B, C;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 545
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (5)
> Testcase: testForeachFRJoin took 0.027 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFRJoin(TestPushDownForeachFlatten.java:619)
> public void testForeachFRJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOJoin frjoin = (LOJoin)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(frjoin).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 619
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(frjoin).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
> }
> comment: same with (1)
> (6)
> Testcase: testForeachFlattenAddedColumnFRJoin took 0.026 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnFRJoin(TestPushDownForeachFlatten.java:738)
> public void testForeachFlattenAddedColumnFRJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 738
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (7)
> Testcase: testForeachInnerJoin took 0.026 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachInnerJoin(TestPushDownForeachFlatten.java:812)
> public void testForeachInnerJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOJoin join = (LOJoin)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(join).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 812
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(join).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
> }
> comment: same with (1)
> (8)
> Testcase: testForeachFlattenAddedColumnInnerJoin took 0.021 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnInnerJoin(TestPushDownForeachFlatten.java:931)
> public void testForeachFlattenAddedColumnInnerJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 931
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> 9.
> [junit] Running org.apache.pig.test.TestTypeCheckingValidator
> [junit] Tests run: 120, Failures: 0, Errors: 1, Time elapsed: 15.047 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in OPEN SOURCE jdk is also correct.
> Detail:
> Testcase: testMapLookupLineage took 0.012 sec
> Caused an ERROR
> org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
> java.lang.ClassCastException: org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
> at org.apache.pig.test.TestTypeCheckingValidator.testMapLookupLineage(TestTypeCheckingValidator.java:5397)
> public void testMapLookupLineage() throws Throwable {
> planTester.buildPlan("a = load 'a' using BinStorage() as (field1, field2: float, field3: chararray );") ;
> planTester.buildPlan("b = foreach a generate field1#'key1' as map1;") ;
> LogicalPlan plan = planTester.buildPlan("c = foreach b generate map1#'key2' + 1 ;") ;
> // validate
> CompilationMessageCollector collector = new CompilationMessageCollector() ;
> TypeCheckingValidator typeValidator = new TypeCheckingValidator() ;
> typeValidator.validate(plan, collector) ;
> printMessageCollector(collector) ;
> printTypeGraph(plan) ;
> planTester.printPlan(plan, TypeCheckingTestUtil.getCurrentMethodName());
> if (collector.hasError()) {
> throw new AssertionError("Expect no error") ;
> }
> LOForEach foreach = (LOForEach)plan.getLeaves().get(0);// see comement1
> LogicalPlan foreachPlan = foreach.getForEachPlans().get(0);
> LogicalOperator exOp = foreachPlan.getRoots().get(0);//see comment2
> // the root would be the project and there would be cast
> // to map between the project and LOMapLookup
> LOCast cast1 = (LOCast)foreachPlan.getSuccessors(exOp).get(0);//line 5397
> assertTrue(cast1.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
> LOMapLookup map = (LOMapLookup)foreachPlan.getSuccessors(cast1).get(0);
> LOCast cast = (LOCast)foreachPlan.getSuccessors(map).get(0);
> assertTrue(cast.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
> }
> comment1:
> foreachPlan initialize process, this will cause the different foreachPlan.
> foreachPlanforeachplan.getLeaves
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> comment2:
> exOpforeachPlan.getRoots(), this is the root cause
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> SUN JDK output:
> foreachPlan:
> Add scope-12 FieldSchema: int Type: int
> |
> |---Const scope-15( 1 ) FieldSchema: int Type: int
> |
> |---Cast scope-19 FieldSchema: int Type: int
> |
> |---MapLookup scope-14 FieldSchema: bytearray Type: bytearray
> |
> |---Cast scope-18 FieldSchema: map Type: map
> |
> |---Project scope-13 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
> Input: b: ForEach scope-6
> getRoots method process:
> keySet() method:
> Cast scope-19
> Cast scope-18
> Add scope-12
> Project scope-13 Projections: [0] Overloaded: false(add to mRoots)
> MapLookup scope-14
> Const scope-15( 1 )(add to mRoots)
> foreachPlan.getRoots():(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
> (Name: Const scope-15( 1 ) Operator Key: scope-15)
> exOp:(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
> OPEN SOURCE JDK output:
> foreachPlan:
> Add scope-13 FieldSchema: int Type: int
> |
> |---Const scope-12( 1 ) FieldSchema: int Type: int
> |
> |---Cast scope-19 FieldSchema: int Type: int
> |
> |---MapLookup scope-15 FieldSchema: bytearray Type: bytearray
> |
> |---Cast scope-18 FieldSchema: map Type: map
> |
> |---Project scope-14 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
> Input: b: ForEach scope-6
> getRoots method process:
> keySet() method:
> Cast scope-18
> Cast scope-19
> Const scope-12( 1 )(add to mRoots)
> Add scope-13
> Project scope-14 Projections: [0] Overloaded: false(add to mRoots)
> MapLookup scope-15
> foreachPlan.getRoots():(Name: Const scope-12( 1 ) Operator Key: scope-12)
> (Name: Project scope-14 Projections: [0] Overloaded: false Operator Key: scope-14)//output in different order compared with SUN JDK output
> exOp:(Name: Const scope-12( 1 ) Operator Key: scope-12)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2281) pig-0.8.1: some unit test case failed
with open source JDK
Posted by "fang fang chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
fang fang chen updated PIG-2281:
--------------------------------
Assignee: fang fang chen
> pig-0.8.1: some unit test case failed with open source JDK
> ----------------------------------------------------------
>
> Key: PIG-2281
> URL: https://issues.apache.org/jira/browse/PIG-2281
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: open source jdk
> ant: 1.8.2
> Reporter: fang fang chen
> Assignee: fang fang chen
> Fix For: 0.8.1
>
>
> Based on my analysis, I think this is because HashMap does not guarantee ordering.
> 1.
> [junit] Running org.apache.pig.test.TestDataModel
> [junit] Tests run: 22, Failures: 1, Errors: 0, Time elapsed: 0.382 sec
> Caused by different output of HashMap.toString() with open source JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in open source jdk is also correct.
> Detail:
> Testcase: testTupleToString took 0.002 sec
> FAILED
> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
> public void testTupleToString() throws Exception {
> Tuple t = giveMeOneOfEach();
> assertEquals("toString", "((3,3.0),{(4),(mary had a little lamb)},[hello#world,goodbye#all],42,5000000000,3.1415927,2.99792458E8,true,hello,goodbye,)", t.toString());//line 269
> }
> comment:
> private Tuple giveMeOneOfEach() throws Exception {
> TupleFactory tf = TupleFactory.getInstance();
> Tuple t1 = tf.newTuple(11);
> Tuple t2 = tf.newTuple(2);
> t2.set(0, new Integer(3));
> t2.set(1, new Float(3.0));
> DataBag bag = BagFactory.getInstance().newDefaultBag();
> bag.add(tf.newTuple(new Integer(4)));
> bag.add(tf.newTuple(new String("mary had a little lamb")));
> Map<String, Object> map = new HashMap<String, Object>(2);
> map.put(new String("hello"), new String("world"));
> map.put(new String("goodbye"), new String("all"));
> t1.set(0, t2);
> t1.set(1, bag);
> t1.set(2, map);//when run t.toString(), HashMap.toString() will be invoked
> t1.set(3, new Integer(42));
> t1.set(4, new Long(5000000000L));
> t1.set(5, new Float(3.141592654));
> t1.set(6, new Double(2.99792458e8));
> t1.set(7, new Boolean(true));
> t1.set(8, new DataByteArray("hello"));
> t1.set(9, new String("goodbye"));
> return t1;
> }
> }
> 2.
> [junit] Running org.apache.pig.test.TestLogToPhyCompiler
> [junit] Tests run: 23, Failures: 1, Errors: 0, Time elapsed: 1.16 sec
> Maybe caused by different output of HashMap.keySet() with open source JDK and sun jdk.
> Detail:
> Failure information:
> Testcase: testSplit took 0.226 sec
> FAILED
> Plan not match
> junit.framework.AssertionFailedError: Plan not match
> at org.apache.pig.test.TestLogToPhyCompiler.testSplit(TestLogToPhyCompiler.java:444)
> public void testSplit() throws VisitorException, IOException {
> String query = "split (load 'a') into x if $0 < '7', y if $0 > '7';";
> LogicalPlan plan = buildPlan(query);
> log.info("ff test plan:"+plan);
> PhysicalPlan pp = buildPhysicalPlan(plan);
> log.info("ff test pp:"+pp);
> int MAX_SIZE = 100000;
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
>
> pp.explain(baos);
> baos.write((int)'\n');
> String compiledPlan = baos.toString();
> compiledPlan = compiledPlan.replaceAll("Load(.*)","Load()");
> if(generate){
> FileOutputStream fos = new FileOutputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
> fos.write(baos.toByteArray());
> return;
> }
>
> FileInputStream fis1 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
> FileInputStream fis2 = new FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split2.gld");
> byte[] b1 = new byte[MAX_SIZE];
> byte[] b2 = new byte[MAX_SIZE];
> int len = fis1.read(b1);
> int test = fis2.read(b2);
> //System.out.println("Length of first plan = " + len + " of second = " + test);
> String goldenPlan1 = new String(b1, 0, len);
> String goldenPlan2 = new String(b2, 0, len);
> goldenPlan1 = goldenPlan1.replaceAll("Load(.*)","Load()");
> goldenPlan2 = goldenPlan2.replaceAll("Load(.*)","Load()");
> System.out.println();
> System.out.println(compiledPlan);
> System.out.println("-------------");
> if(compiledPlan.compareTo(goldenPlan1) == 0 || compiledPlan.compareTo(goldenPlan2) == 0) {
> // good
> }
> else {
> System.out.println("Expected plan1=") ;
> System.out.println(goldenPlan1) ;
> System.out.println("Expected plan2=") ;
> System.out.println(goldenPlan1) ;
> System.out.println("Actual plan=") ;
> System.out.println(compiledPlan) ;
> System.out.println("**END**") ;
> fail("Plan not match") ;//line 444
> }
>
> }
> comment:
> variable compiledPlan initialize invoke the following methods:
> pp.explain(baos);// explain(OutputStream out) method
> baos.write((int)'\n');
> String compiledPlan = baos.toString();
> explain(OutputStream out)explain(OutputStream out, boolean verbose)print(OutputStream printer)depthFirstPP()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> baos variable output:
> > open source jdk:
> > x: Filter[tuple] - Test-Plan-Builder-240
> > | |
> > | Less Than[boolean] - Test-Plan-Builder-243
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-241
> > | |
> > | |---Constant(7) - Test-Plan-Builder-242
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> >
> > y: Filter[tuple] - Test-Plan-Builder-244
> > | |
> > | Greater Than[boolean] - Test-Plan-Builder-247
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-245
> > | |
> > | |---Constant(7) - Test-Plan-Builder-246
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> > sun jdk:
> > y: Filter[tuple] - Test-Plan-Builder-240
> > | |
> > | Greater Than[boolean] - Test-Plan-Builder-243
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-241
> > | |
> > | |---Constant(7) - Test-Plan-Builder-242
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> >
> > x: Filter[tuple] - Test-Plan-Builder-244
> > | |
> > | Less Than[boolean] - Test-Plan-Builder-247
> > | |
> > | |---Project[bytearray][0] - Test-Plan-Builder-245
> > | |
> > | |---Constant(7) - Test-Plan-Builder-246
> > |
> > |---Split - Test-Plan-Builder-239
> > |
> > |---229: Load()
> 3.
> [junit] Running org.apache.pig.test.TestMRCompiler
> [junit] Tests run: 25, Failures: 1, Errors: 0, Time elapsed: 0.729 sec
> Maybe, caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> Detail:
> Testcase: testSortUDF1 took 0.02 sec
> FAILED
> null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
> | ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> junit.framework.ComparisonFailure: null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
> | ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
> at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
> private void run(PhysicalPlan pp, String expectedFile) throws Exception {
> String compiledPlan, goldenPlan = null;
> int MAX_SIZE = 100000;
> MRCompiler comp = new MRCompiler(pp, pc);
> comp.compile();
> MROperPlan mrp = comp.getMRPlan();
> PlanPrinter ppp = new PlanPrinter(mrp);
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
> ppp.print(baos);//see "comments"
> compiledPlan = baos.toString();//compiledPlan's initialize is based on baos
> if(generate ){
> FileOutputStream fos = new FileOutputStream(expectedFile);
> fos.write(baos.toByteArray());
> return;
> }
> FileInputStream fis = new FileInputStream(expectedFile);
> byte[] b = new byte[MAX_SIZE];
> int len = fis.read(b);
> goldenPlan = new String(b, 0, len);
> if (goldenPlan.charAt(len-1) == '\n')
> goldenPlan = goldenPlan.substring(0, len-1);
> pp.explain(System.out);
> System.out.println();
> System.out.println("<<<" + compiledPlan + ">>>");
> System.out.println("-------------");
> System.out.println("Golden");
> System.out.println("<<<" + goldenPlan + ">>>");
> System.out.println("-------------");
> assertEquals(goldenPlan, compiledPlan);//line 1056
> }
> comment:
> ppp.print(baos) invokes method as following:
> print(OutputStream printer) depthFirstPP()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> 4.
> [junit] Running org.apache.pig.test.TestMergeJoinOuter
> [junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 132.66 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> Testcase: testCompilation took 0.443 sec
> FAILED
> junit.framework.AssertionFailedError:
> at org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116)
> Iterator<MapReduceOper> itr = mrPlan.iterator();// see comments
> MapReduceOper oper = itr.next();
> assertTrue(oper.reducePlan.isEmpty());//line 116
> comments:
> iterator() method:
> public Iterator<E> iterator() {
> return mOps.keySet().iterator();//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> }
> with same mrPlan:
> MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
> Reduce Plan Empty
> | C: Store(hdfs://localhost.localdomain:34390/user/root/out:org.apache.pig.builtin.PigStorage) - scope-38
> | |
> | |---C: New For Each(true,true)[tuple] - scope-37
> | | |
> | | Project[bag][1] - scope-31
> | | |
> | | POBinCond[bag] - scope-36
> | | |
> | | |---Project[bag][2] - scope-32
> | | |
> | | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - scope-34
> | | | |
> | | | |---Project[bag][2] - scope-33
> | | |
> | | |---Constant({(,,)}) - scope-35
> | |
> | |---C: MergeCogroup[tuple] - scope-30
> | |
> | |---A: Load(hdfs://localhost.localdomain:34390/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader) - scope-22
> |
> |---MapReduce(1) - scope-41:
> | Store(hdfs://localhost.localdomain:34390/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage) - scope-48
> | |
> | |---New For Each(true)[bag] - scope-47
> | | |
> | | Project[tuple][1] - scope-46
> | |
> | |---Package[tuple]{tuple} - scope-45
> | Local Rearrange[tuple]{tuple}(false) - scope-44
> | | |
> | | Project[tuple][*] - scope-43
> | |
> variable itr is in different order:
> open source jdk:
> (Name: MapReduce(1) - scope-41(itr.next())
> (Name: MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) - scope-39:
> sun jdk:
> (Name: MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) - scope-39: (itr.next())
> (Name: MapReduce(1) - scope-41:
> 5.
> [junit] Running org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor
> [junit] Tests run: 25, Failures: 3, Errors: 0, Time elapsed: 1.081 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1).
> Testcase: testSimplePlan took 0.295 sec
> FAILED
> expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
> junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testSimplePlan(TestNewPlanLogToPhyTranslationVisitor.java:131)
> public void testSimplePlan() throws Exception {
> LogicalPlanTester lpt = new LogicalPlanTester(pc);
> lpt.buildPlan("a = load 'd.txt';");
> lpt.buildPlan("b = filter a by $0==NULL;");
> LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
>
> org.apache.pig.newplan.logical.relational.LogicalPlan newLogicalPlan = migratePlan(plan);
> PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
>
> assertEquals( 3, phyPlan.size() );
> assertEquals( 1, phyPlan.getRoots().size() );
> assertEquals( 1, phyPlan.getLeaves().size() );
>
> PhysicalOperator load = phyPlan.getRoots().get(0);
> assertEquals( POLoad.class, load.getClass() );
> assertTrue( ((POLoad)load).getLFile().getFileName().contains("d.txt") );
>
> // Check for Filter
> PhysicalOperator fil = phyPlan.getSuccessors(load).get(0);
> assertEquals( POFilter.class, fil.getClass() );
> PhysicalPlan filPlan = ((POFilter)fil).getPlan();
> assertEquals( 2, filPlan.getRoots().size() );
> assertEquals( 1, filPlan.getLeaves().size() );
>
> PhysicalOperator eq = filPlan.getLeaves().get(0);
> assertEquals( EqualToExpr.class, eq.getClass() );
>
> PhysicalOperator prj1 = filPlan.getRoots().get(0);
> assertEquals( POProject.class, prj1.getClass() );//line 131
> assertEquals( 0, ((POProject)prj1).getColumn() );
> PhysicalOperator constExp = filPlan.getRoots().get(1);
> assertEquals( ConstantExpression.class, constExp.getClass() );
> assertEquals( null, ((ConstantExpression)constExp).getValue() );
>
> // Check for Store
> PhysicalOperator stor = phyPlan.getSuccessors(fil).get(0);
> assertEquals( POStore.class, stor.getClass() );
> assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty"));
> }
> comment:
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
>
> (2).
> Testcase: testJoinPlan took 0.062 sec
> FAILED
> expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
> junit.framework.AssertionFailedError: expected:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression> but was:<class org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testJoinPlan(TestNewPlanLogToPhyTranslationVisitor.java:201)
> public void testJoinPlan() throws Exception {
> LogicalPlanTester lpt = new LogicalPlanTester(pc);
> lpt.buildPlan("a = load 'd1.txt' as (id, c);");
> lpt.buildPlan("b = load 'd2.txt'as (id, c);");
> lpt.buildPlan("c = join a by id, b by c;");
> lpt.buildPlan("d = filter c by a::id==NULL AND b::c==NULL;");
> LogicalPlan plan = lpt.buildPlan("store d into 'empty';");
>
> // check basics
> org.apache.pig.newplan.logical.relational.LogicalPlan newPlan = migratePlan(plan);
> PhysicalPlan physicalPlan = translatePlan(newPlan);
> assertEquals(9, physicalPlan.size());
> assertEquals(physicalPlan.getRoots().size(), 2);
>
> // Check Load and LocalRearrange and GlobalRearrange
> PhysicalOperator LoR = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(0)).get(0);
> assertEquals( POLocalRearrange.class, LoR.getClass() );
> POLocalRearrange Lor = (POLocalRearrange) LoR;
> PhysicalOperator prj3 = Lor.getPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj3.getClass() );
> assertEquals(0, ((POProject)prj3).getColumn() );
> PhysicalOperator inp1 = Lor.getInputs().get(0);
> assertEquals( POLoad.class, inp1.getClass() );
> assertTrue( ((POLoad)inp1).getLFile().getFileName().contains("d1.txt") );
>
> PhysicalOperator LoR1 = (PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(1)).get(0);
> assertEquals( POLocalRearrange.class, LoR1.getClass() );
> POLocalRearrange Lor1 = (POLocalRearrange) LoR1;
> PhysicalOperator prj4 = Lor1.getPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj4.getClass() );
> assertEquals(1, ((POProject)prj4).getColumn() );
> PhysicalOperator inp2 = Lor1.getInputs().get(0);
> assertEquals( POLoad.class, inp2.getClass() );
> assertTrue( ((POLoad)inp2).getLFile().getFileName().contains("d2.txt") );
>
> PhysicalOperator GoR = (PhysicalOperator)physicalPlan.getSuccessors(LoR).get(0);
> assertEquals( POGlobalRearrange.class, GoR.getClass() );
>
> PhysicalOperator Pack = (PhysicalOperator)physicalPlan.getSuccessors(GoR).get(0);
> assertEquals( POPackage.class, Pack.getClass() );
> // Check for ForEach
> PhysicalOperator ForE = (PhysicalOperator)physicalPlan.getSuccessors(Pack).get(0);
> assertEquals( POForEach.class, ForE.getClass() );
> PhysicalOperator prj5 = ((POForEach)ForE).getInputPlans().get(0).getLeaves().get(0);
> assertEquals( POProject.class, prj5.getClass() );
> assertEquals( 1, ((POProject)prj5).getColumn() );
> PhysicalOperator prj6 = ((POForEach)ForE).getInputPlans().get(1).getLeaves().get(0);
> assertEquals( POProject.class, prj6.getClass() );
> assertEquals( 2, ((POProject)prj6).getColumn() );
>
> // Filter Operator
> PhysicalOperator fil = (PhysicalOperator)physicalPlan.getSuccessors(ForE).get(0);
> assertEquals( POFilter.class, fil.getClass() );
>
> PhysicalPlan filPlan = ((POFilter)fil).getPlan();
> List<PhysicalOperator> filRoots = filPlan.getRoots();
>
> assertEquals( ConstantExpression.class, filRoots.get(1).getClass() );//line 201
> ConstantExpression ce1 = (ConstantExpression) filRoots.get(1);
> assertEquals( null, ce1.getValue() );
> assertEquals( ConstantExpression.class, filRoots.get(3).getClass() );
> ConstantExpression ce2 = (ConstantExpression) filRoots.get(3);
> assertEquals( null, ce2.getValue() );
> assertEquals( POProject.class, filRoots.get(0).getClass() );
> POProject prj1 = (POProject) filRoots.get(0);
> assertEquals( 3, prj1.getColumn() );
> assertEquals( POProject.class, filRoots.get(2).getClass() );
> POProject prj2 = (POProject) filRoots.get(2);
> assertEquals( 0, prj2.getColumn() );
> // Check Store Operator
> PhysicalOperator stor = (PhysicalOperator)physicalPlan.getSuccessors(fil).get(0);
> assertEquals( POStore.class, stor.getClass() );
> assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty") );
> }
> comment:
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> (3).
> Testcase: testMultiStore took 0.083 sec
> FAILED
> expected:<1> but was:<0>
> junit.framework.AssertionFailedError: expected:<1> but was:<0>
> at org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testMultiStore(TestNewPlanLogToPhyTranslationVisitor.java:255)
> PhysicalOperator prj2 = Lor1.getPlans().get(0).getLeaves().get(0);
> assertEquals(1, ((POProject)prj2).getColumn() );//line 255
> comment:
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> 6.
> [junit] Running org.apache.pig.test.TestPruneColumn
> [junit] Tests run: 67, Failures: 4, Errors: 0, Time elapsed: 528.047 sec
> Maybe caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1).
> Testcase: testMapKey2 took 6.291 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKey2(TestPruneColumn.java:1213)
> public void testMapKey2() throws Exception{
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
> pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");// see comment3
> pigServer.registerQuery("C = foreach B generate $0#'key2', $1;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");//see comment1
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("2"));
> assertTrue(t.get(1).toString().equals("1"));
>
> assertTrue(iter.hasNext());
> t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("4"));
> assertTrue(t.get(1).toString().equals("2"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0",
> "Map key required for A: $1->[key2, key1]"}));// line 1213 see comment2
> }
> comments1:
> pigServer.openIterator("C") invokes the following method to save pigServer information to filename:
> store(String id, String filename, String func)
> comment2:
> public boolean checkLogFileMessage(String[] messages)
> {
> BufferedReader reader = null;
>
> try {
> reader = new BufferedReader(new FileReader(logFile));//logFile=filename
> List<String> logMessages=new ArrayList<String>();
> String line;
> while ((line=reader.readLine())!=null)
> {
> logMessages.add(line);
> }
>
> // Check if all messages appear in the log
> for (int i=0;i<messages.length;i++)
> {
> boolean found = false;
> for (int j=0;j<logMessages.size();j++)
> if (logMessages.get(j).contains(messages[i])) {
> found = true;
> break;
> }
> if (!found)
> return false;
> }
>
> // Check no other log besides messages
> for (int i=0;i<logMessages.size();i++) {
> boolean found = false;
> for (int j=0;j<messages.length;j++) {
> if (logMessages.get(i).contains(messages[j])) {
> found = true;
> break;
> }
> }
> if (!found) {
> if (logMessages.get(i).contains("Columns pruned for")||
> logMessages.get(i).contains("Map key required for")) {
> return false;
> }
> }
> }
> return true;
> }
> catch (IOException e) {
> return false;
> }
> }
> comment3:
> the content in filename is different, because pigServer(invoke HashMap.keySet()) is different.
> pigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> (2)
> Testcase: testMapKey3 took 6.319 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKey3(TestPruneColumn.java:1229)
> public void testMapKey3() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as (a0:int, a1:map[]);");
> pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");
> pigServer.registerQuery("C = group B all;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("all"));
> assertTrue(t.get(1).toString().equals("{([key2#2,key1#1],1),([key2#4,key1#2],2)}"));
> //line 1229
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0"}));
> }
> comment:
> variable "t" initialize process:
> titerpigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> different t.get(1).toString():
> OPEN SOURCE JDK:
> {([key1#1,key2#2],1),([key1#2,key2#4],2)}
> SUN JDK:
> {([key2#2,key1#1],1),([key2#4,key1#2],2)}
> (3).
> Testcase: testMapKeyInSplit1 took 6.3 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testMapKeyInSplit1(TestPruneColumn.java:1303)
> public void testMapKeyInSplit1() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as (m:map[]);");
> pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
> pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
> pigServer.registerQuery("D = join B by key1, C by key2;");
>
> Iterator<Tuple> iter = pigServer.openIterator("D");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.size()==2);
> assertTrue(t.get(0).toString().equals("2"));
> assertTrue(t.get(1).toString().equals("2"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Map key required for A: $0->[key2, key1]"}));//line 1303
> }
> comment: same with (1).
> (4).
> Testcase: testSharedSchemaObject took 6.327 sec
> FAILED
> null
> junit.framework.AssertionFailedError: null
> at org.apache.pig.test.TestPruneColumn.testSharedSchemaObject(TestPruneColumn.java:1626)
> public void testSharedSchemaObject() throws Exception {
> pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile10.toString(), pigServer.getPigContext()) + "' AS (a0, a1:map[], a2);");
> pigServer.registerQuery("B = foreach A generate a1;");
> pigServer.registerQuery("C = limit B 10;");
>
> Iterator<Tuple> iter = pigServer.openIterator("C");
>
> assertTrue(iter.hasNext());
> Tuple t = iter.next();
> assertTrue(t.toString().equals("([2#1,1#1])"));
>
> assertFalse(iter.hasNext());
>
> assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0, $2"}));// line 1626
> }
> comment: same with (2).
> 7.
> [junit] Running org.apache.pig.test.TestUnionOnSchema
> [junit] Tests run: 21, Failures: 1, Errors: 0, Time elapsed: 196.841 sec
> Testcase: testUnionOnSchemaScopedColumnNameNeg took 0.008 sec
> FAILED
> Expected exception message matching 'Found more than one match: l1::i, l2::i' but got 'Error during parsing. Found more than one match: l2::i, l1::i'
> at org.apache.pig.test.TestUnionOnSchema.checkSchemaEx(TestUnionOnSchema.java:604)
> at org.apache.pig.test.TestUnionOnSchema.testUnionOnSchemaScopedColumnNameNeg(TestUnionOnSchema.java:370)
> 8..
> [junit] Running org.apache.pig.test.TestPushDownForeachFlatten
> [junit] Tests run: 37, Failures: 0, Errors: 8, Time elapsed: 1.455 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk.
> (1)
> Testcase: testForeachUnion took 0.039 sec
> Caused an ERROR
> Expected LOForEach, got LOUnion
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOUnion
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachUnion(TestPushDownForeachFlatten.java:275)
> public void testForeachUnion() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> LogicalPlan lp = planTester.buildPlan("D = union B, C;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
>
> LOLoad load = (LOLoad) lp.getRoots().get(0);// see comment
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 275
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment:
> LOLoad load = (LOLoad) lp.getRoots().get(0);//
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> (2)
> Testcase: testForeachCogroup took 0.038 sec
> Caused an ERROR
> Expected LOForEach, got LOCogroup
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCogroup
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCogroup(TestPushDownForeachFlatten.java:295)
> public void testForeachCogroup() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> LogicalPlan lp = planTester.buildPlan("D = cogroup B by $0, C by $0;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
>
> LOLoad load = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 295
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (3)
> Testcase: testForeachCross took 0.035 sec
> Caused an ERROR
> Expected LOForEach, got LOCross
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachCross(TestPushDownForeachFlatten.java:427)
> public void testForeachCross() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> planTester.buildPlan("D = cross B, C;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOCross cross = (LOCross)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(cross).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));//line 427
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(cross).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
>
> }
> comment: same with (1)
> (4)
> Testcase: testForeachFlattenAddedColumnCross took 0.034 sec
> Caused an ERROR
> Expected LOForEach, got LOCross
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOCross
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnCross(TestPushDownForeachFlatten.java:545)
> public void testForeachFlattenAddedColumnCross() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = cross B, C;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 545
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (5)
> Testcase: testForeachFRJoin took 0.027 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFRJoin(TestPushDownForeachFlatten.java:619)
> public void testForeachFRJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference);");
> planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOJoin frjoin = (LOJoin)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(frjoin).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 619
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(frjoin).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
> }
> comment: same with (1)
> (6)
> Testcase: testForeachFlattenAddedColumnFRJoin took 0.026 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnFRJoin(TestPushDownForeachFlatten.java:738)
> public void testForeachFlattenAddedColumnFRJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 738
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> comment: same with (1)
> (7)
> Testcase: testForeachInnerJoin took 0.026 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachInnerJoin(TestPushDownForeachFlatten.java:812)
> public void testForeachInnerJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad load = (LOLoad) lp.getRoots().get(0);
> LOLimit limit = (LOLimit) lp.getLeaves().get(0);
> LOJoin join = (LOJoin)lp.getPredecessors(limit).get(0);
> LOForEach foreach = (LOForEach) lp.getPredecessors(join).get(0);
>
> Schema limitSchema = limit.getSchema();
>
> assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 812
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == true);
> assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
> pushDownForeach.transform(lp.getSuccessors(load));
>
> planTester.rebuildSchema(lp);
>
> for(Boolean b: foreach.getFlatten()) {
> assertEquals(b.booleanValue(), false);
> }
>
> LOForEach newForeach = (LOForEach)lp.getSuccessors(join).get(0);
>
>
> List<Boolean> newForeachFlatten = newForeach.getFlatten();
> Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
> for(Integer key: remap.keySet()) {
> Integer value = remap.get(key);
> assertEquals(newForeachFlatten.get(value).booleanValue(), true);
> }
>
> assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
> }
> comment: same with (1)
> (8)
> Testcase: testForeachFlattenAddedColumnInnerJoin took 0.021 sec
> Caused an ERROR
> Expected LOForEach, got LOJoin
> org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected LOForEach, got LOJoin
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
> at org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
> at org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnInnerJoin(TestPushDownForeachFlatten.java:931)
> public void testForeachFlattenAddedColumnInnerJoin() throws Exception {
> planTester.buildPlan("A = load 'myfile' as (name, age, gpa:(letter_grade, point_score));");
> planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
> planTester.buildPlan("C = load 'anotherfile' as (name, age, preference:(course_name, instructor));");
> planTester.buildPlan("D = join B by $0, C by $0;");
> LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
>
> planTester.setPlan(lp);
> planTester.setProjectionMap(lp);
> planTester.rebuildSchema(lp);
>
> PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
> LOLoad loada = (LOLoad) lp.getRoots().get(0);
>
> assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 931
> assertTrue(pushDownForeach.getSwap() == false);
> assertTrue(pushDownForeach.getInsertBetween() == false);
> assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
> }
> 9.
> [junit] Running org.apache.pig.test.TestTypeCheckingValidator
> [junit] Tests run: 120, Failures: 0, Errors: 1, Time elapsed: 15.047 sec
> Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun jdk. Based on the discussion with Thejas Nair(pig committer) the output in OPEN SOURCE jdk is also correct.
> Detail:
> Testcase: testMapLookupLineage took 0.012 sec
> Caused an ERROR
> org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
> java.lang.ClassCastException: org.apache.pig.impl.logicalLayer.LOAdd incompatible with org.apache.pig.impl.logicalLayer.LOCast
> at org.apache.pig.test.TestTypeCheckingValidator.testMapLookupLineage(TestTypeCheckingValidator.java:5397)
> public void testMapLookupLineage() throws Throwable {
> planTester.buildPlan("a = load 'a' using BinStorage() as (field1, field2: float, field3: chararray );") ;
> planTester.buildPlan("b = foreach a generate field1#'key1' as map1;") ;
> LogicalPlan plan = planTester.buildPlan("c = foreach b generate map1#'key2' + 1 ;") ;
> // validate
> CompilationMessageCollector collector = new CompilationMessageCollector() ;
> TypeCheckingValidator typeValidator = new TypeCheckingValidator() ;
> typeValidator.validate(plan, collector) ;
> printMessageCollector(collector) ;
> printTypeGraph(plan) ;
> planTester.printPlan(plan, TypeCheckingTestUtil.getCurrentMethodName());
> if (collector.hasError()) {
> throw new AssertionError("Expect no error") ;
> }
> LOForEach foreach = (LOForEach)plan.getLeaves().get(0);// see comement1
> LogicalPlan foreachPlan = foreach.getForEachPlans().get(0);
> LogicalOperator exOp = foreachPlan.getRoots().get(0);//see comment2
> // the root would be the project and there would be cast
> // to map between the project and LOMapLookup
> LOCast cast1 = (LOCast)foreachPlan.getSuccessors(exOp).get(0);//line 5397
> assertTrue(cast1.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
> LOMapLookup map = (LOMapLookup)foreachPlan.getSuccessors(cast1).get(0);
> LOCast cast = (LOCast)foreachPlan.getSuccessors(map).get(0);
> assertTrue(cast.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
> }
> comment1:
> foreachPlan initialize process, this will cause the different foreachPlan.
> foreachPlanforeachplan.getLeaves
> public List<E> getLeaves() {
> if (mLeaves.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) { //mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mFromEdges.get(op) == null) {
> mLeaves.add(op);
> }
> }
> }
> return mLeaves;
> }
> comment2:
> exOpforeachPlan.getRoots(), this is the root cause
> public List<E> getRoots() {
> if (mRoots.size() == 0 && mOps.size() > 0) {
> for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
> if (mToEdges.get(op) == null) {
> mRoots.add(op);
> }
> }
> }
> return mRoots;
> }
> SUN JDK output:
> foreachPlan:
> Add scope-12 FieldSchema: int Type: int
> |
> |---Const scope-15( 1 ) FieldSchema: int Type: int
> |
> |---Cast scope-19 FieldSchema: int Type: int
> |
> |---MapLookup scope-14 FieldSchema: bytearray Type: bytearray
> |
> |---Cast scope-18 FieldSchema: map Type: map
> |
> |---Project scope-13 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
> Input: b: ForEach scope-6
> getRoots method process:
> keySet() method:
> Cast scope-19
> Cast scope-18
> Add scope-12
> Project scope-13 Projections: [0] Overloaded: false(add to mRoots)
> MapLookup scope-14
> Const scope-15( 1 )(add to mRoots)
> foreachPlan.getRoots():(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
> (Name: Const scope-15( 1 ) Operator Key: scope-15)
> exOp:(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key: scope-13)
> OPEN SOURCE JDK output:
> foreachPlan:
> Add scope-13 FieldSchema: int Type: int
> |
> |---Const scope-12( 1 ) FieldSchema: int Type: int
> |
> |---Cast scope-19 FieldSchema: int Type: int
> |
> |---MapLookup scope-15 FieldSchema: bytearray Type: bytearray
> |
> |---Cast scope-18 FieldSchema: map Type: map
> |
> |---Project scope-14 Projections: [0] Overloaded: false FieldSchema: map1: bytearray Type: bytearray
> Input: b: ForEach scope-6
> getRoots method process:
> keySet() method:
> Cast scope-18
> Cast scope-19
> Const scope-12( 1 )(add to mRoots)
> Add scope-13
> Project scope-14 Projections: [0] Overloaded: false(add to mRoots)
> MapLookup scope-15
> foreachPlan.getRoots():(Name: Const scope-12( 1 ) Operator Key: scope-12)
> (Name: Project scope-14 Projections: [0] Overloaded: false Operator Key: scope-14)//output in different order compared with SUN JDK output
> exOp:(Name: Const scope-12( 1 ) Operator Key: scope-12)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira