[jira] [Commented] (PIG-2700) Unit tests fail against Hadoop 2.0.0

["Cheolsoo Park (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276459#comment-13276459 ] 

Cheolsoo Park commented on PIG-2700:
------------------------------------

Please ignore PIG-2460. It was uploaded accidentally.
                
> Unit tests fail against Hadoop 2.0.0
> ------------------------------------
>
>                 Key: PIG-2700
>                 URL: https://issues.apache.org/jira/browse/PIG-2700
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.3
>            Reporter: Cheolsoo Park
>         Attachments: PIG-2460.patch, PIG-2700.patch
>
>
> I am running Pig unit tests against Hadoop 2.0.0-SNAPSHOT as follows:
> {code}
> --- ivy/libraries.properties
> +++ ivy/libraries.properties
> @@ -37,9 +37,9 @@ guava.version=11.0
>  jersey-core.version=1.8
>  hadoop-core.version=1.0.0
>  hadoop-test.version=1.0.0
> -hadoop-common.version=0.23.1
> -hadoop-hdfs.version=0.23.1
> -hadoop-mapreduce.version=0.23.1
> +hadoop-common.version=2.0.0-SNAPSHOT
> +hadoop-hdfs.version=2.0.0-SNAPSHOT
> +hadoop-mapreduce.version=2.0.0-SNAPSHOT
> {code}
> And I see the following issues:
> 1) copyFromLocalToCluster fails:
> {code}
> fs command '-put AccumulatorInput.txt AccumulatorInput.txt' failed. Please check output logs for details
> java.io.IOException: fs command '-put AccumulatorInput.txt AccumulatorInput.txt' failed. Please check output logs for details
>     at org.apache.pig.tools.grunt.GruntParser.processFsCommand(GruntParser.java:1012)
> {code}
> I am getting around this problem by explicitly creating intermediate directories that do not exist. (Please see the attached patch.)
> 2) Many tests including TestAccumulator hang and eventually timeout. The JVM thread dump shows the following call stack:
> {code}
> [junit]    java.lang.Thread.State: TIMED_WAITING (sleeping)
> [junit]     at java.lang.Thread.sleep(Native Method)
> [junit]     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:245)
> [junit]     at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
> [junit]     at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
> [junit]     at org.apache.pig.PigServer.storeEx(PigServer.java:996)
> [junit]     at org.apache.pig.PigServer.store(PigServer.java:963)
> [junit]     at org.apache.pig.PigServer.openIterator(PigServer.java:876)
> [junit]     at org.apache.pig.test.TestAccumulator.testAccumBasic(TestAccumulator.java:150)
> {code}
> This is because test jobs are never finished in the mini cluster. The reason why test jobs are never finished is because they fail with a ClassNotFound exception while being executed.
> In fact, this is a regression of HADOOP-6963 where hadoop introduced dependency on Apache Commons IO library:
> {code:title=FileUtil.java}
> isSymLink = org.apache.commons.io.FileUtils.isSymlink(allFiles[i]);
> {code}
> But the Apache Commons IO library is missing in Pig, so test jobs keep failing in the mini cluster until timeout.
> I am fixing this issue by adding commons-io-2.3.jar to ivy.xml and library.properties. (Please see the attached patch.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira