You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2012/07/22 23:03:35 UTC
[jira] [Commented] (PIG-2791) Pig does not work with Namenode
Federation
[ https://issues.apache.org/jira/browse/PIG-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420330#comment-13420330 ]
Rohini Palaniswamy commented on PIG-2791:
-----------------------------------------
In addition to PIG-2791-2.patch, this patch ontains fix for unit tests for hadoop 23. Even after including Cheolsoo's patch from PIG-2700, there were lot of unit test failures.
1) MiniYarnCluster node manager would not start with guice errors due to to jersey-guice-1.8 and guice-2.0 incompatibility. Took 2 days to figure this one out. Had to change dependency to guice-3.0 and add few more dependencies.
2) FSShell copy command behaviour has changed. Had to do mkdir before copy commands as copy commands do not create the destination directory structure if it does not exist. Also had to add creation of fs working directory (user home directory in dfs) while creating MiniCluster.
Ran full suite of unit and e2e tests for hadoop 23 with branch 10.
> Pig does not work with Namenode Federation
> ------------------------------------------
>
> Key: PIG-2791
> URL: https://issues.apache.org/jira/browse/PIG-2791
> Project: Pig
> Issue Type: Bug
> Components: grunt
> Affects Versions: 0.10.0
> Environment: Pig QE
> Reporter: patrick white
> Assignee: Rohini Palaniswamy
> Priority: Blocker
> Attachments: PIG-2791-0.patch, PIG-2791-1.patch, PIG-2791-2.patch, PIG-2791-3-branch10.patch, PIG-2791-3-trunk.patch, asf_test_notes.txt
>
>
> The Yahoo Pig QE team ran into a blocking issue when trying to test Client-Side Mount Tables, on a Federated cluster with two NNs, this blocks Pig Testing on Federation.
> Federation relies strongly on the use of CSMT with viewFS, QE found that in this configuration it is not possible to enter grunt shell because Pig makes a call to getDefaultReplication() on the fs, which is ambiguous over viewFS and causes core to throw a org.apache.hadoop.fs.viewfs.NotInMountpointException: "getDefaultReplication on empty path is invalid".
> This in turn cause Pig to exit with an internal error as follows:
> 2012-07-06 22:20:25,657 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.1.0.1206081058 (r1348169) compiled Jun 08 2012, 17:58:42
> 2012-07-06 22:20:26,074 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 2012-07-06 22:20:26,076 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: viewfs:///
> 2012-07-06 22:20:26,080 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - ERROR 2999: Unexpected internal error. getDefaultReplication on empty path is invalid
> 2012-07-06 22:20:26,522 [main] WARN org.apache.pig.Main - There is no log file to write to.
> 2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:482)
> at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:77)
> at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:205)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:118)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:208)
> at org.apache.pig.PigServer.<init>(PigServer.java:246)
> at org.apache.pig.PigServer.<init>(PigServer.java:231)
> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
> at org.apache.pig.Main.run(Main.java:487)
> at org.apache.pig.Main.main(Main.java:111)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira