You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by lulynn_2008 <lu...@163.com> on 2011/08/17 04:56:18 UTC
about pig-0.8.1 unit test
Hello,
I runTestDataModel test case with IBM JDK, and got the following error:
Testcase: testTupleToString took 0.002 sec
FAILED
toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
This is because IBM HashMap.toString() output different with SUN HashMap.toString(). My question is:
Should the output must be the one expected?
Are the output expected based on SUN JDK?
Does the failed test case mean pig-0.8.1 can not work with IBM JDk?
Thank you.
Re:Re: about pig-0.8.1 unit test
Posted by lulynn_2008 <lu...@163.com>.
Hello,
For test case TestPruneColumn:
public void testMapKeyInSplit1() throws Exception {
pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as (m:map[]);");
pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
pigServer.registerQuery("D = join B by key1, C by key2;");
Iterator<Tuple> iter = pigServer.openIterator("D");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("2"));
assertTrue(t.get(1).toString().equals("2"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Map key required for A: $0->[key2, key1]"}));
}
I have some questions:
1. What content will be stored in logFile in methond checkLogFileMessage?
2. With other JDK but not SUN JDK, I found the variable item contain string like "Map key required for A: $0->[key1, key2]". I think this is caused by pigServer.registerQuery method which invoke method parse. And parse method is based on HashMap structure.Could you please tell me what HashMap methods will be invoked and cause the different output LogicalPlan in parse method?
public LogicalPlan parse(String scope,
String query,
Map<LogicalOperator, LogicalPlan> aliases,
Map<OperatorKey, LogicalOperator> opTable,
Map<String, LogicalOperator> aliasOp,
int start,
Map<String, String> fileNameMap)
throws IOException, ParseException {
ByteArrayInputStream in = new ByteArrayInputStream(query.getBytes());
QueryParser parser = new QueryParser(in, pigContext, scope, aliases, opTable,
aliasOp, start, fileNameMap);
return parser.Parse();
}
3.The same situation also happened in:
Testcase: testMapKey2 took 8.133 sec
Should be:"Columns pruned for A: $0",
"Map key required for A: $1->[key2, key1]"
But is:"Columns pruned for A: $0"
"Map key required for A: $1->[key1, key2]"
Testcase: testMapKey3 took 8.815 sec
Should be:{([key2#2,key1#1],1),([key2#4,key1#2],2)}
But is:{([key1#1,key2#2],1),([key1#2,key2#4],2)}
Testcase: testSharedSchemaObject took 9.15 sec
Should be:([2#1,1#1])
But is:([1#1,2#1])
Are these results also correct?
At 2011-08-19 01:50:20,"Thejas Nair" <th...@hortonworks.com> wrote:
>yes, even in this case, both outputs look correct.
>-Thejas
>
>
>
>On 8/18/11 6:05 AM, lulynn_2008 wrote:
>> Thejas Nair,
>>
>> Thank you
>> Actually there are other unit tests failed because of different order
>> output. During parse the pig command, a HashMap variable "aliases" will
>> be used in
>> src-gen/org/apache/pig/impl/logicalLayer/parser/QueryParser.java. I
>> think this unit test failure is also caused by HashMap problem. What is
>> your opinion?
>> Here are two failure unit test examples:
>> TestLogToPhyCompiler: Query: split (load 'a') into x if $0 < '7', y if
>> $0 > '7';
>> Testcase: testSplit took 0.198 sec
>> FAILED
>> IBM JDK:
>> x: Filter[tuple] - Test-Plan-Builder-240
>> | |
>> | Less Than[boolean] - Test-Plan-Builder-243
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-241
>> | |
>> | |---Constant(7) - Test-Plan-Builder-242
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>>
>> y: Filter[tuple] - Test-Plan-Builder-244
>> | |
>> | Greater Than[boolean] - Test-Plan-Builder-247
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-245
>> | |
>> | |---Constant(7) - Test-Plan-Builder-246
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>> SUN JDK:
>> y: Filter[tuple] - Test-Plan-Builder-240
>> | |
>> | Greater Than[boolean] - Test-Plan-Builder-243
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-241
>> | |
>> | |---Constant(7) - Test-Plan-Builder-242
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>>
>> x: Filter[tuple] - Test-Plan-Builder-244
>> | |
>> | Less Than[boolean] - Test-Plan-Builder-247
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-245
>> | |
>> | |---Constant(7) - Test-Plan-Builder-246
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>> TestMRCompiler(IBM JDK):
>> Testcase: testSortUDF1 took 0.02 sec
>> FAILED
>> null
>> expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator])
>> - -18:
>> | ...> but
>> was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
>> | ...>
>> junit.framework.ComparisonFailure: null
>> expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator])
>> - -18:
>> | ...> but
>> was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
>> | ...>
>> at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
>> at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
>>
>>
>>
>>
>> At 2011-08-18 01:17:02,"Thejas Nair" <th...@hortonworks.com> wrote:
>>>Since HashMap does not guarantee ordering, this test is not guaranteed
>>>to pass, ie the test is faulty and it needs to be fixed. The result from
>>>IBM JDK is also correct.
>>>An easy fix for the test case would be to test for both sequences of
>>>hash-map.
>>>Can you please open a jira ?
>>>
>>>-Thejas
>>>
>>>
>>>
>>>On 8/16/11 7:56 PM, lulynn_2008 wrote:
>>>> Hello,
>>>> I runTestDataModel test case with IBM JDK, and got the following error:
>>>> Testcase: testTupleToString took 0.002 sec
>>>> FAILED
>>>> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>>>> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>>>> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
>>>> This is because IBM HashMap.toString() output different with SUN HashMap.toString(). My question is:
>>>> Should the output must be the one expected?
>>>> Are the output expected based on SUN JDK?
>>>> Does the failed test case mean pig-0.8.1 can not work with IBM JDk?
>>>> Thank you.
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
Re: about pig-0.8.1 unit test
Posted by Thejas Nair <th...@hortonworks.com>.
yes, even in this case, both outputs look correct.
-Thejas
On 8/18/11 6:05 AM, lulynn_2008 wrote:
> Thejas Nair,
>
> Thank you
> Actually there are other unit tests failed because of different order
> output. During parse the pig command, a HashMap variable "aliases" will
> be used in
> src-gen/org/apache/pig/impl/logicalLayer/parser/QueryParser.java. I
> think this unit test failure is also caused by HashMap problem. What is
> your opinion?
> Here are two failure unit test examples:
> TestLogToPhyCompiler: Query: split (load 'a') into x if $0 < '7', y if
> $0 > '7';
> Testcase: testSplit took 0.198 sec
> FAILED
> IBM JDK:
> x: Filter[tuple] - Test-Plan-Builder-240
> | |
> | Less Than[boolean] - Test-Plan-Builder-243
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-241
> | |
> | |---Constant(7) - Test-Plan-Builder-242
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
>
> y: Filter[tuple] - Test-Plan-Builder-244
> | |
> | Greater Than[boolean] - Test-Plan-Builder-247
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-245
> | |
> | |---Constant(7) - Test-Plan-Builder-246
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
> SUN JDK:
> y: Filter[tuple] - Test-Plan-Builder-240
> | |
> | Greater Than[boolean] - Test-Plan-Builder-243
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-241
> | |
> | |---Constant(7) - Test-Plan-Builder-242
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
>
> x: Filter[tuple] - Test-Plan-Builder-244
> | |
> | Less Than[boolean] - Test-Plan-Builder-247
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-245
> | |
> | |---Constant(7) - Test-Plan-Builder-246
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
> TestMRCompiler(IBM JDK):
> Testcase: testSortUDF1 took 0.02 sec
> FAILED
> null
> expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator])
> - -18:
> | ...> but
> was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> junit.framework.ComparisonFailure: null
> expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator])
> - -18:
> | ...> but
> was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
> | ...>
> at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
> at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
>
>
>
>
> At 2011-08-18 01:17:02,"Thejas Nair" <th...@hortonworks.com> wrote:
>>Since HashMap does not guarantee ordering, this test is not guaranteed
>>to pass, ie the test is faulty and it needs to be fixed. The result from
>>IBM JDK is also correct.
>>An easy fix for the test case would be to test for both sequences of
>>hash-map.
>>Can you please open a jira ?
>>
>>-Thejas
>>
>>
>>
>>On 8/16/11 7:56 PM, lulynn_2008 wrote:
>>> Hello,
>>> I runTestDataModel test case with IBM JDK, and got the following error:
>>> Testcase: testTupleToString took 0.002 sec
>>> FAILED
>>> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>>> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>>> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
>>> This is because IBM HashMap.toString() output different with SUN HashMap.toString(). My question is:
>>> Should the output must be the one expected?
>>> Are the output expected based on SUN JDK?
>>> Does the failed test case mean pig-0.8.1 can not work with IBM JDk?
>>> Thank you.
>>>
>>>
>>>
>>
>
>
>
Re:Re: about pig-0.8.1 unit test
Posted by lulynn_2008 <lu...@163.com>.
Thejas Nair,Thank you
Actually there are other unit tests failed because of different order output. During parse the pig command, a HashMap variable "aliases" will be used in src-gen/org/apache/pig/impl/logicalLayer/parser/QueryParser.java. I think this unit test failure is also caused by HashMap problem. What is your opinion?
Here are two failure unit test examples:
TestLogToPhyCompiler:Query: split (load 'a') into x if $0 < '7', y if $0 > '7';
Testcase: testSplit took 0.198 sec
FAILED
IBM JDK:
x: Filter[tuple] - Test-Plan-Builder-240
| |
| Less Than[boolean] - Test-Plan-Builder-243
| |
| |---Project[bytearray][0] - Test-Plan-Builder-241
| |
| |---Constant(7) - Test-Plan-Builder-242
|
|---Split - Test-Plan-Builder-239
|
|---229: Load()
y: Filter[tuple] - Test-Plan-Builder-244
| |
| Greater Than[boolean] - Test-Plan-Builder-247
| |
| |---Project[bytearray][0] - Test-Plan-Builder-245
| |
| |---Constant(7) - Test-Plan-Builder-246
|
|---Split - Test-Plan-Builder-239
|
|---229: Load()
SUN JDK:
y: Filter[tuple] - Test-Plan-Builder-240
| |
| Greater Than[boolean] - Test-Plan-Builder-243
| |
| |---Project[bytearray][0] - Test-Plan-Builder-241
| |
| |---Constant(7) - Test-Plan-Builder-242
|
|---Split - Test-Plan-Builder-239
|
|---229: Load()
x: Filter[tuple] - Test-Plan-Builder-244
| |
| Less Than[boolean] - Test-Plan-Builder-247
| |
| |---Project[bytearray][0] - Test-Plan-Builder-245
| |
| |---Constant(7) - Test-Plan-Builder-246
|
|---Split - Test-Plan-Builder-239
|
|---229: Load()
TestMRCompiler(IBM JDK):
Testcase: testSortUDF1 took 0.02 sec
FAILED
null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
| ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
| ...>
junit.framework.ComparisonFailure: null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
| ...> but was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
| ...>
at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
At 2011-08-18 01:17:02,"Thejas Nair" <th...@hortonworks.com> wrote:
>Since HashMap does not guarantee ordering, this test is not guaranteed
>to pass, ie the test is faulty and it needs to be fixed. The result from
>IBM JDK is also correct.
>An easy fix for the test case would be to test for both sequences of
>hash-map.
>Can you please open a jira ?
>
>-Thejas
>
>
>
>On 8/16/11 7:56 PM, lulynn_2008 wrote:
>> Hello,
>> I runTestDataModel test case with IBM JDK, and got the following error:
>> Testcase: testTupleToString took 0.002 sec
>> FAILED
>> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
>> This is because IBM HashMap.toString() output different with SUN HashMap.toString(). My question is:
>> Should the output must be the one expected?
>> Are the output expected based on SUN JDK?
>> Does the failed test case mean pig-0.8.1 can not work with IBM JDk?
>> Thank you.
>>
>>
>>
>
Re: about pig-0.8.1 unit test
Posted by Thejas Nair <th...@hortonworks.com>.
Since HashMap does not guarantee ordering, this test is not guaranteed
to pass, ie the test is faulty and it needs to be fixed. The result from
IBM JDK is also correct.
An easy fix for the test case would be to test for both sequences of
hash-map.
Can you please open a jira ?
-Thejas
On 8/16/11 7:56 PM, lulynn_2008 wrote:
> Hello,
> I runTestDataModel test case with IBM JDK, and got the following error:
> Testcase: testTupleToString took 0.002 sec
> FAILED
> toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> junit.framework.ComparisonFailure: toString expected:<...ad a little lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
> at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
> This is because IBM HashMap.toString() output different with SUN HashMap.toString(). My question is:
> Should the output must be the one expected?
> Are the output expected based on SUN JDK?
> Does the failed test case mean pig-0.8.1 can not work with IBM JDk?
> Thank you.
>
>
>