You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Valentin Logvinskiy (JIRA)" <ji...@apache.org> on 2016/12/27 22:55:58 UTC
[jira] [Comment Edited] (HADOOP-13939) hadoop jar -libjars not
working with mainClass argument
[ https://issues.apache.org/jira/browse/HADOOP-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15781458#comment-15781458 ]
Valentin Logvinskiy edited comment on HADOOP-13939 at 12/27/16 10:55 PM:
-------------------------------------------------------------------------
[~templedf] Yes, just rechecked.
{code}
public class MainClass extends Configured implements Tool {
private static final Logger log = LoggerFactory.getLogger(MainClass.class);
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(HBaseConfiguration.create(), new MainClass(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
Configuration conf = this.getConf();
log.info(conf.get("tmpjars"));
return 0;
}
}
{code}
{noformat}
hadoop jar file.jar MainClass -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:21,252 INFO EventsToJourneysJobMain:69 -
{noformat}
{noformat}
hadoop jar file.jar -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:42,677 INFO EventsToJourneysJobMain:69 - file:/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
{noformat}
More information with debug file GenericOptionsParser on line:
{code}
this.commandLine = parser.parse(opts, this.preProcessForWindows(args), true); //version 2.7.2 from maven repo
{code}
With MainClass:
{noformat}
commandLine = {CommandLine@1337}
args = {LinkedList@1343} size = 3
0 = "MainClass"
1 = "-libjars"
2 = "/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar"
options = {ArrayList@1344} size = 0
{noformat}
Without MainClass argument:
{noformat}
commandLine = {CommandLine@1337}
args = {LinkedList@1342} size = 0
options = {ArrayList@1343} size = 1
0 = {Option@1345} "[ option: libjars [ARG] :: comma separated jar files to include in the classpath. ]"
opt = "libjars"
longOpt = null
argName = "paths"
description = "comma separated jar files to include in the classpath."
required = false
optionalArg = false
numberOfArgs = 1
type = null
values = {ArrayList@1350} size = 1
valuesep = '\u0000' 0
{noformat}
was (Author: gron1987):
[~templedf] Yes, just recheck.
{code}
public class MainClass extends Configured implements Tool {
private static final Logger log = LoggerFactory.getLogger(MainClass.class);
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(HBaseConfiguration.create(), new MainClass(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
Configuration conf = this.getConf();
log.info(conf.get("tmpjars"));
return 0;
}
}
{code}
{noformat}
hadoop jar file.jar MainClass -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:21,252 INFO EventsToJourneysJobMain:69 -
{noformat}
{noformat}
hadoop jar file.jar -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:42,677 INFO EventsToJourneysJobMain:69 - file:/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
{noformat}
More information with debug file GenericOptionsParser on line:
{code}
this.commandLine = parser.parse(opts, this.preProcessForWindows(args), true); //version 2.7.2 from maven repo
{code}
With MainClass:
{noformat}
commandLine = {CommandLine@1337}
args = {LinkedList@1343} size = 3
0 = "MainClass"
1 = "-libjars"
2 = "/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar"
options = {ArrayList@1344} size = 0
{noformat}
Without MainClass argument:
{noformat}
commandLine = {CommandLine@1337}
args = {LinkedList@1342} size = 0
options = {ArrayList@1343} size = 1
0 = {Option@1345} "[ option: libjars [ARG] :: comma separated jar files to include in the classpath. ]"
opt = "libjars"
longOpt = null
argName = "paths"
description = "comma separated jar files to include in the classpath."
required = false
optionalArg = false
numberOfArgs = 1
type = null
values = {ArrayList@1350} size = 1
valuesep = '\u0000' 0
{noformat}
> hadoop jar -libjars not working with mainClass argument
> -------------------------------------------------------
>
> Key: HADOOP-13939
> URL: https://issues.apache.org/jira/browse/HADOOP-13939
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.7.2
> Reporter: Valentin Logvinskiy
>
> If we have
> {code:java}
> public class MainClass extends Configured implements Tool {
> public static void main(String[] args) throws Exception {
> ToolRunner.run(conf, new MainClass(), args);
> }
>
> ...
>
> @Override
> public int run(String[] args) throws Exception {
> ...
> }
> }
> {code}
> Running this code with -libjars argument in distributed mode will not work.
> {noformat}
> hadoop jar file.jar MainClass -libjars /path/to/file
> {noformat}
> This not working due to error in GenericOptionsParser
> GenericOptionsParser.java:
> {code}
> commandLine = parser.parse(opts, args, true);
> {code}
> As far as 3rd argument is "true" this mean:
> {noformat}
> * @param stopAtNonOption specifies whether to continue parsing the
> * arguments if a non option is encountered.
> {noformat}
> Our run arguments are:
> {noformat}
> MainClass -libjars /path/to/file
> {noformat}
> MainClass is nonOption and after this argument everything else don't start parsing.
> If we have only arguments then processGeneralOptions method will return nothing and will not add tmpjars config.
> Changing to
> {noformat}
> commandLine = parser.parse(opts, args, false);
> {noformat}
> works for me, but could affect a lot of developers (in theory)
> Fast fix - do not use mainClass argument at all.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org