[jira] [Commented] (PIG-4939) QueryParserUtils.setHdfsServers(QueryParserUtils.java:104) should not be called for non-dfs methods

["Adam Szita (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15632679#comment-15632679 ] 

Adam Szita commented on PIG-4939:
---------------------------------

Hi [~daijy], although it would be quicker to hardcode "hbase" scheme to be handled differently in getRemoteHosts, I think we should rather go with the marker interface option. From an OO point of view it is a cleaner solution - to distinguish LoadFunc implementations that aren't file system related. This is also safer in my opinion, but of course in the case above PhoenixHBaseLoader has to be modified too in order to make use of this.
I attached patch for the Pig change [^PIG-4939.patch] and also one for Phoenix just for illustration [^PIG-4939-PHOENIX.patch]. 

> QueryParserUtils.setHdfsServers(QueryParserUtils.java:104) should not be called for non-dfs methods
> ---------------------------------------------------------------------------------------------------
>
>                 Key: PIG-4939
>                 URL: https://issues.apache.org/jira/browse/PIG-4939
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Siddhi Mehta
>            Assignee: Adam Szita
>            Priority: Minor
>         Attachments: PIG-4939-PHOENIX.patch, PIG-4939.patch
>
>
> {code}
> A = load 'hbase://query/SELECT ID,NAME,DATE FROM HIRES WHERE DATE > TO_DATE('1990-12-21 05:55:00.000');
> STORE A into 'output';
> {code}
> The above script throws an exception because it treats the location as an fs path and tries to convert it to a URI after splitting it based on comma.
> The code that tries to the same is 
> {code}
>  String buildLoadOp(SourceLocation loc, String alias, String filename, FuncSpec funcSpec, LogicalSchema schema)
>     throws ParserValidationException {
>         String absolutePath;
>         LoadFunc loFunc;
>         try {
>             // Load LoadFunc class from default properties if funcSpec is null. Fallback on PigStorage if LoadFunc is not specified in properties.
>             funcSpec = funcSpec == null ? new FuncSpec(pigContext.getProperties().getProperty(PigConfiguration.PIG_DEFAULT_LOAD_FUNC, PigStorage.class.getName())) : funcSpec;
>             loFunc = (LoadFunc)PigContext.instantiateFuncFromSpec(funcSpec);
>             ......
>             .......
>             if (absolutePath == null) {
>                 absolutePath = loFunc.relativeToAbsolutePath( filename, QueryParserUtils.getCurrentDir( pigContext ) );
>                 if (absolutePath!=null) {
>                     QueryParserUtils.setHdfsServers( absolutePath, pigContext );
>                 }
>          .....  
>        }
> {code}
> We should not be calling QueryParserUtils.setHdfsServers(QueryParserUtils.java:104) should not be called for non-dfs methods



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)