You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Eric Hanson (JIRA)" <ji...@apache.org> on 2014/03/04 21:06:25 UTC
[jira] [Updated] (HIVE-6546) WebHCat job submission for pig with
-useHCatalog argument fails on Windows
[ https://issues.apache.org/jira/browse/HIVE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Hanson updated HIVE-6546:
------------------------------
Description:
On a one-box windows setup, do the following from a powershell prompt:
cmd /c curl.exe -s `
-d user.name=hadoop `
-d arg=-useHCatalog `
-d execute="emp = load '/data/emp/emp_0.dat'; dump emp;" `
-d statusdir="/tmp/webhcat.output01" `
'http://localhost:50111/templeton/v1/pig' -v
The job fails with error code 7, but it should run.
I traced this down to the following. In the job configuration for the TempletonJobController, we have templeton.args set to
cmd,/c,call,C:\\hadoop\\\\pig-0.11.0.1.3.0.0-0846/bin/pig.cmd,-D__WEBHCAT_TOKEN_FILE_LOCATION__="-useHCatalog",-execute,"emp = load '/data/emp/emp_0.dat'; dump emp;"
Notice the = sign before "-useHCatalog". I think this should be a comma.
The bad string D__WEBHCAT_TOKEN_FILE_LOCATION__="-useHCatalog" gets created in org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows().
It happens at line 434:
{code}
} else {
if (i < args.length - 1) {
prop += "=" + args[++i]; // RIGHT HERE! at iterations i = 37, 38
}
}
{code}
Bug is here:
{code}
if (prop != null) {
if (prop.contains("=")) { // -D__WEBHCAT_TOKEN_FILE_LOCATION__ does not contain equal, so else branch is run and appends ="-useHCatalog",
// everything good
} else {
if (i < args.length - 1) {
prop += "=" + args[++i];
}
}
newArgs.add(prop);
}
{code}
One possible fix is to change the string constant org.apache.hcatalog.templeton.tool.TempletonControllerJob.TOKEN_FILE_ARG_PLACEHOLDER to have an "=" sign in it. Or, preProcessForWindows() itself could be changed.
> WebHCat job submission for pig with -useHCatalog argument fails on Windows
> --------------------------------------------------------------------------
>
> Key: HIVE-6546
> URL: https://issues.apache.org/jira/browse/HIVE-6546
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
> Affects Versions: 0.11.0, 0.12.0, 0.13.0
> Environment: Windows Azure HDINSIGHT and Windows one-box installations.
> Reporter: Eric Hanson
>
> On a one-box windows setup, do the following from a powershell prompt:
> cmd /c curl.exe -s `
> -d user.name=hadoop `
> -d arg=-useHCatalog `
> -d execute="emp = load '/data/emp/emp_0.dat'; dump emp;" `
> -d statusdir="/tmp/webhcat.output01" `
> 'http://localhost:50111/templeton/v1/pig' -v
> The job fails with error code 7, but it should run.
> I traced this down to the following. In the job configuration for the TempletonJobController, we have templeton.args set to
> cmd,/c,call,C:\\hadoop\\\\pig-0.11.0.1.3.0.0-0846/bin/pig.cmd,-D__WEBHCAT_TOKEN_FILE_LOCATION__="-useHCatalog",-execute,"emp = load '/data/emp/emp_0.dat'; dump emp;"
> Notice the = sign before "-useHCatalog". I think this should be a comma.
> The bad string D__WEBHCAT_TOKEN_FILE_LOCATION__="-useHCatalog" gets created in org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows().
> It happens at line 434:
> {code}
> } else {
> if (i < args.length - 1) {
> prop += "=" + args[++i]; // RIGHT HERE! at iterations i = 37, 38
> }
> }
> {code}
> Bug is here:
> {code}
> if (prop != null) {
> if (prop.contains("=")) { // -D__WEBHCAT_TOKEN_FILE_LOCATION__ does not contain equal, so else branch is run and appends ="-useHCatalog",
> // everything good
> } else {
> if (i < args.length - 1) {
> prop += "=" + args[++i];
> }
> }
> newArgs.add(prop);
> }
> {code}
> One possible fix is to change the string constant org.apache.hcatalog.templeton.tool.TempletonControllerJob.TOKEN_FILE_ARG_PLACEHOLDER to have an "=" sign in it. Or, preProcessForWindows() itself could be changed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)