[jira] [Commented] (PIG-5025) Improve TestLoad.java: use own separated folder under /tmp

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489298#comment-15489298 ] 

liyunzhang_intel commented on PIG-5025:
---------------------------------------

[~szita] and [~daijy]:   I also found that TestLoad#testCommaSeparatedString2 and TestLoad#testGlobChars sometimes fail both in trunk and spark branch jenkins.  This troubles me sometimes because all unit tests pass except these two thus the jenkins result fail(actually no regression unit test failures)

From [~szita]'s comment, it seems that the problem is because the working directory is set as hdfs:/tmp/ and TestLoad tries to load all files matching "t?s*,test". If hadoop system generates a file naming "t:2sTest.txt"  in hdfs:/tmp, hdfs:/tmp/t:2sTest.txt will be loaded and URISyntaxException is thrown out.  In the patch, set the working directory as /tmp/usr/pig to avoid to load hdfs:/tmp/t:2sTest.txt.



> Improve TestLoad.java: use own separated folder under /tmp
> ----------------------------------------------------------
>
>                 Key: PIG-5025
>                 URL: https://issues.apache.org/jira/browse/PIG-5025
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>            Priority: Minor
>         Attachments: PIG-5025.patch
>
>
> Test cases testCommaSeparatedString2 and testGlobChars may fail if for some reason files (from any other sources) in /tmp have : (colon) in the filenames. This is because HDFS doesn't support colon since it has its own URI handling. Exception below.
> I propose we separate the working dir of these tests to use their own folder in /tmp.
> Failed to parse: java.net.URISyntaxException: Relative path in absolute URI: t:2sTest.txt
> 	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
> 	at org.apache.pig.test.TestLoad.checkLoadPath(TestLoad.java:317)
> 	at org.apache.pig.test.TestLoad.checkLoadPath(TestLoad.java:299)
> 	at org.apache.pig.test.TestLoad.testCommaSeparatedString2(TestLoad.java:189)
> Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: t:2sTest.txt
> 	at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> 	at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> 	at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> 	at org.apache.hadoop.fs.Globber.doGlob(Globber.java:260)
> 	at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> 	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1637)
> 	at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asCollection(HDataStorage.java:215)
> 	at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asCollection(HDataStorage.java:41)
> 	at org.apache.pig.builtin.JsonMetadata.findMetaFile(JsonMetadata.java:119)
> 	at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> 	at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:518)
> 	at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
> 	at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
> 	at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:866)
> 	at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
> 	at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
> 	at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
> 	at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
> 	at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
> 	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: t:2sTest.txt
> 	at java.net.URI.checkPath(URI.java:1823)
> 	at java.net.URI.<init>(URI.java:745)
> 	at org.apache.hadoop.fs.Path.initialize(Path.java:203)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)