You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by Ethan Xu <et...@gmail.com> on 2016/02/05 02:51:16 UTC
Compatibility with MR1 Cloudera cdh4.2.1
Hello,
I got an error when running the systemML/scripts/Univar-Stats.dml script on
a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
the bottom of the email. The same script ran fine on a smaller sample
(several MB) of the same data set, when MR was not invoked.
The main error was java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble()
Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
the JobConf.getDouble() method.
The hadoop-core jar of CDH4.2.1 can be found here:
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
The calling line of SystemML is line 1194 of
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
I was wondering, if the finding is accurate, is there a potential fix, or
does this mean the current version of SystemML is not compatible with
CDH4.2.1?
Thank you,
Ethan
hadoop jar $sysDir/target/SystemML.jar -f
$sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
X=$baseDirHDFS/original-coded.csv
TYPES=$baseDirHDFS/original-coded-type.csv
STATS=$baseDirHDFS/univariate-summary.csv
16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
(./SystemML-config.xml) found
16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
text read for 'text' and 'csv' due to thread contention on JRE < 1.8
(java.version=1.7.0_71).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
Total execution time: 0.880 sec.
Number of executed MR Jobs: 0.
16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
at
org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
at org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
at
org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
at
org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Shirish Tatikonda <sh...@gmail.com>.
Hi Ethan,
The getDouble() method is actually part of
org.apache.hadoop.conf.Configuration.java, which is part of hadoop-common
but not hadoop-core -- see [1]. Seems like, it used to be part of
hadoop-core a long time ago.
Also, the pom.xml in SystemML project does specify hadoop-common as the
dependency (as provided library). Make sure hadoop-common jar file is in
the classpath.
[1]
http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.3.0/org/apache/hadoop/conf/Configuration.java
Shirish
On Thu, Feb 4, 2016 at 5:51 PM, Ethan Xu <et...@gmail.com> wrote:
> Hello,
>
> I got an error when running the systemML/scripts/Univar-Stats.dml script on
> a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
> the bottom of the email. The same script ran fine on a smaller sample
> (several MB) of the same data set, when MR was not invoked.
>
> The main error was java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble()
> Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
> the JobConf.getDouble() method.
>
> The hadoop-core jar of CDH4.2.1 can be found here:
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>
> The calling line of SystemML is line 1194 of
>
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>
> I was wondering, if the finding is accurate, is there a potential fix, or
> does this mean the current version of SystemML is not compatible with
> CDH4.2.1?
>
> Thank you,
>
> Ethan
>
>
> hadoop jar $sysDir/target/SystemML.jar -f
> $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> X=$baseDirHDFS/original-coded.csv
> TYPES=$baseDirHDFS/original-coded-type.csv
> STATS=$baseDirHDFS/univariate-summary.csv
>
> 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> (./SystemML-config.xml) found
> 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
> 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.7.0_71).
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> Total execution time: 0.880 sec.
> Number of executed MR Jobs: 0.
>
> 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> at
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> at
> org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> at
>
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> at
>
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Matthias Boehm <mb...@us.ibm.com>.
Thanks for letting us know Ethan. The reason was likely that
JobContext.TASK_ISMAP was not in the configuration.
We'll setup an MR v1 cluster next week and systematically resolve all these
incompatibility issues in order to avoid additional round trips.
Regards,
Matthias
From: Ethan Xu <et...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 02/05/2016 08:04 PM
Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
Seems it's a problem of
String taskType = (conf.getBoolean(JobContext.TASK_ISMAP, true)) ? "m" :
"r"
;
on line 137 of
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter
conf.getBoolean(JobContext.TASK_ISMAP, true) returns 'true' (perhaps using
provided default value) even in reducers
causing (line 139) 'charIx' to get value -1 that leads to the error
Confirmed on my test case. When error occurs in reducer, 'name' has value
'0-r-00000', but 'taskType' has value 'm'.
The following (ugly) hack fixed the problem. DML ran successfully
afterwards.
Change line 137 - 139 of
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter
to:
String name = file.getName();
Pattern p = Pattern.compile("-[rm]-");
Matcher m = p.matcher(name);
int charIx = 0;
if (m.find()) {
charIx = m.start();
}else{
throw new RuntimeException("file name :" + name + "doesn't contain 'r'
or 'm'");
}
Ethan
On Fri, Feb 5, 2016 at 4:37 PM, Ethan Xu <et...@gmail.com> wrote:
> Thanks tried that and moved a bit further. Now a new exception (still in
> reduce phase of 'CSV-Reblock-MR'):
>
> WARN org.apache.hadoop.mapred.Child: Error running child
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1911)
> at
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.moveFileToDestination
(MultipleOutputCommitter.java:140)
> at
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.moveFinalTaskOutputs
(MultipleOutputCommitter.java:119)
> at
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.commitTask
(MultipleOutputCommitter.java:94)
> at org.apache.hadoop.mapred.OutputCommitter.commitTask
(OutputCommitter.java:221)
> at org.apache.hadoop.mapred.Task.commit(Task.java:1005)
> at org.apache.hadoop.mapred.Task.done(Task.java:875)
> at org.apache.hadoop.mapred.ReduceTask.run
(ReduceTask.java:453)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs
(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
>
>
> On Fri, Feb 5, 2016 at 4:03 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
>
>> ok that is interesting. I think the following is happening: The hadoop
>> version is >2.0, which makes SystemML switch to the 2.x configuration
>> properties. However, because MR1 is bundled into this distribution these
>> configurations do not exist which makes us fail on processing task ids.
>>
>> Workaround: Change
>> org.apache.sysml.runtime.matrix.mapred.MRConfigurationNames line 85 to
>> *"boolean* hadoopVersion2 = false".
>>
>> Regards,
>> Matthias
>>
>> [image: Inactive hide details for Ethan Xu ---02/05/2016 12:36:27
>> PM---Thank you very much. I just pulled the update, rebuilt the
proje]Ethan
>> Xu ---02/05/2016 12:36:27 PM---Thank you very much. I just pulled the
>> update, rebuilt the project and reran the code.
>>
>> From: Ethan Xu <et...@gmail.com>
>> To: dev@systemml.incubator.apache.org
>> Date: 02/05/2016 12:36 PM
>> Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
>> ------------------------------
>>
>>
>>
>> Thank you very much. I just pulled the update, rebuilt the project and
>> reran the code.
>>
>> The method-not-found error was gone, and the MapReduce job was kicked
off.
>> The 'Assign-RowID-MR' job finished successfully.
>> The map phase of 'CSV-Reblock-MR' job finished, but reducers threw
>> NullPointerExceptions at
>>
>> java.lang.NullPointerException
>> at
>> org.apache.sysml.runtime.matrix.mapred.ReduceBase.close
(ReduceBase.java:205)
>> at org.apache.hadoop.mapred.ReduceTask.runOldReducer
(ReduceTask.java:516)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs
(UserGroupInformation.java:1408)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>
>> The job I ran was the same as before on the same data:
>> hadoop jar <SystemML dir>/target/SystemML.jar -libjars <local
>> dir>/hadoop-lzo-0.4.15.jar -f <SystemML
>> dir>/scripts/algorithms/Univar-Stats.dml -nvargs X=<HDFS
>> dir>/original-coded.csv TYPES=<HDFS dir>/original-coded-type.csv
>> STATS=<HDFS dir>/univariate-summary.csv
>>
>> The hadoop cluster was also the same one: CDH4.2.1.
>>
>> Sorry for keep coming back with problems on a really old hadoop system.
>> Please let me know what other information is needed to diagnose the
issue.
>>
>> Ethan
>>
>>
>> On Fri, Feb 5, 2016 at 1:26 PM, Deron Eriksson <de...@gmail.com>
>> wrote:
>>
>> > Hi Ethan,
>> >
>> > I believe your safest, cleanest bet is to wait for the fix from
>> Matthias.
>> > When he pushes the fix, you will see it at
>> > https://github.com/apache/incubator-systemml/commits/master. At that
>> > point,
>> > you can pull (git pull) the changes from GitHub to your machine and
then
>> > build with Maven utilizing the new changes.
>> >
>> > Alternatively, it's not really recommended, but you might be able to
use
>> > -libjars to reference the hadoop-commons jar, which should be in your
>> local
>> > maven repository
>> >
>> >
>>
(.m2/repository/org/apache/hadoop/hadoop-common/2.4.1/hadoop-common-2.4.1.jar).
>> > However, mixing jar versions usually doesn't work very well (it can
>> lead to
>> > other problems), so waiting for the fix is best.
>> >
>> > Deron
>> >
>> >
>> > On Fri, Feb 5, 2016 at 6:47 AM, Ethan Xu <et...@gmail.com>
>> wrote:
>> >
>> > > Thank you Shirish and Deron for the suggestions. Looking forward to
>> the
>> > fix
>> > > from Matthias!
>> > >
>> > > We are using the hadoop-common shipped with CDH4.2.1, and it's in
>> > > classpath. I'm a bit hesitate to alter our hadoop configuration to
>> > include
>> > > other versions since other people are using it too.
>> > >
>> > > Not sure if/how the following naive approach affects the program
>> > behavior,
>> > > but I did try changing the scope of
>> > >
>> > > <groupId>org.apache.hadoop</groupId>
>> > > <artifactId>hadoop-common</artifactId>
>> > > <version>${hadoop.version}</version>
>> > >
>> > > in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the
jar
>> > > (21MB), and it threw the same error.
>> > >
>> > > By the way this is in pom.xml line 65 - 72:
>> > > <properties>
>> > > <hadoop.version>2.4.1</hadoop.version>
>> > > <antlr.version>4.3</antlr.version>
>> > > <spark.version>1.4.1</spark.version>
>> > >
>> > > <!-- OS-specific JVM arguments for running
integration
>> > > tests -->
>> > > <integrationTestExtraJVMArgs />
>> > > </properties>
>> > >
>> > > Am I supposed to modify the hadoop.version before build?
>> > >
>> > > Thanks again,
>> > >
>> > > Ethan
>> > >
>> > >
>> > >
>> > > On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson <
>> deroneriksson@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Matthias,
>> > > >
>> > > > Glad to hear the fix is simple. Mixing jar versions sometimes is
not
>> > very
>> > > > fun.
>> > > >
>> > > > Deron
>> > > >
>> > > >
>> > > > On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm
<mb...@us.ibm.com>
>> > > wrote:
>> > > >
>> > > > > well, let's not mix different hadoop versions in the class path
or
>> > > > > client/server. If I'm not mistaken, cdh 4.x always shipped with
MR
>> > v1.
>> > > > It's
>> > > > > a trivial fix for us and will be in the repo tomorrow morning
>> anyway.
>> > > > > Thanks for catching this issue Ethan.
>> > > > >
>> > > > > Regards,
>> > > > > Matthias
>> > > > >
>> > > > > [image: Inactive hide details for Deron Eriksson ---02/04/2016
>> > 11:04:38
>> > > > > PM---Hi Ethan, Just FYI, I looked at
>> hadoop-common-2.0.0-cdh4.2]Deron
>> > > > > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I
looked
>> at
>> > > > > hadoop-common-2.0.0-cdh4.2.1.jar (
>> > > > >
>> > > > > From: Deron Eriksson <de...@gmail.com>
>> > > > > To: dev@systemml.incubator.apache.org
>> > > > > Date: 02/04/2016 11:04 PM
>> > > > > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
>> > > > > ------------------------------
>> > > > >
>> > > > >
>> > > > >
>> > > > > Hi Ethan,
>> > > > >
>> > > > > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
>> > > > > ),
>> > > > > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
>> > > > > org.apache.hadoop.conf.Configuration class in that jar doesn't
>> appear
>> > > to
>> > > > > have a getDouble method, so using that version of hadoop-common
>> won't
>> > > > work.
>> > > > >
>> > > > > However, the hadoop-common-2.4.1.jar (
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
>> > > > > )
>> > > > >
>> > > > > does appear to have the getDouble method. It's possible that
>> adding
>> > > that
>> > > > > jar to your classpath may fix your problem, as Shirish pointed
>> out.
>> > > > >
>> > > > > It sounds like Matthias may have another fix.
>> > > > >
>> > > > > Deron
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm
<mboehm@us.ibm.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > well, we did indeed not run on MR v1 for a while now. However,
I
>> > > don't
>> > > > > > want to get that far and say we don't support it anymore. I'll
>> fix
>> > > this
>> > > > > > particular issue by tomorrow.
>> > > > > >
>> > > > > > In the next couple of weeks we should run our full performance
>> > > > testsuite
>> > > > > > (for broad coverage) over an MR v1 cluster and systematically
>> > remove
>> > > > > > unnecessary incompatibility like this instance. Any
volunteers?
>> > > > > >
>> > > > > > Regards,
>> > > > > > Matthias
>> > > > > >
>> > > > > > [image: Inactive hide details for Ethan Xu ---02/04/2016
>> 05:51:28
>> > > > > > PM---Hello, I got an error when running the
>> > > > > systemML/scripts/Univar-S]Ethan
>> > > > > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when
>> running
>> > the
>> > > > > > systemML/scripts/Univar-Stats.dml script on
>> > > > > >
>> > > > > > From: Ethan Xu <et...@gmail.com>
>> > > > > > To: dev@systemml.incubator.apache.org
>> > > > > > Date: 02/04/2016 05:51 PM
>> > > > > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
>> > > > > > ------------------------------
>> > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Hello,
>> > > > > >
>> > > > > > I got an error when running the
>> systemML/scripts/Univar-Stats.dml
>> > > > script
>> > > > > on
>> > > > > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error
>> > message
>> > > > is
>> > > > > at
>> > > > > > the bottom of the email. The same script ran fine on a smaller
>> > sample
>> > > > > > (several MB) of the same data set, when MR was not invoked.
>> > > > > >
>> > > > > > The main error was java.lang.NoSuchMethodError:
>> > > > > > org.apache.hadoop.mapred.JobConf.getDouble()
>> > > > > > Digging deeper, it looks like the CDH4.2.1 version of MR
indeed
>> > > didn't
>> > > > > have
>> > > > > > the JobConf.getDouble() method.
>> > > > > >
>> > > > > > The hadoop-core jar of CDH4.2.1 can be found here:
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>> > > > >
>> > > > > >
>> > > > > > The calling line of SystemML is line 1194 of
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>> > > > > >
>> > > > > > I was wondering, if the finding is accurate, is there a
>> potential
>> > > fix,
>> > > > or
>> > > > > > does this mean the current version of SystemML is not
compatible
>> > with
>> > > > > > CDH4.2.1?
>> > > > > >
>> > > > > > Thank you,
>> > > > > >
>> > > > > > Ethan
>> > > > > >
>> > > > > >
>> > > > > > hadoop jar $sysDir/target/SystemML.jar -f
>> > > > > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
>> > > > > > X=$baseDirHDFS/original-coded.csv
>> > > > > > TYPES=$baseDirHDFS/original-coded-type.csv
>> > > > > > STATS=$baseDirHDFS/univariate-summary.csv
>> > > > > >
>> > > > > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016
>> > > 20:35:03
>> > > > > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
>> > > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML
>> config
>> > > file
>> > > > > > (./SystemML-config.xml) found
>> > > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings
in
>> > > > > DMLConfig
>> > > > > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable
>> > > multi-threaded
>> > > > > > text read for 'text' and 'csv' due to thread contention on JRE
<
>> > 1.8
>> > > > > > (java.version=1.7.0_71).
>> > > > > > SLF4J: Class path contains multiple SLF4J bindings.
>> > > > > > SLF4J: Found binding in
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > > > > > SLF4J: Found binding in
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > > > > > SLF4J: Found binding in
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
>> for
>> > an
>> > > > > > explanation.
>> > > > > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
>> > > > > > Total execution time: 0.880 sec.
>> > > > > > Number of executed MR Jobs: 0.
>> > > > > >
>> > > > > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016
>> > 20:35:07
>> > > > > > Exception in thread "main" java.lang.NoSuchMethodError:
>> > > > > > org.apache.hadoop.mapred.JobConf.getDouble
(Ljava/lang/String;D)D
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1195)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1129)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:307)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:289)
>> > > > > > at
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob
(CSVReblockMR.java:275)
>> > > > > > at
>> > > > >
>> org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob
(RunMRJobs.java:143)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction
(MRJobInstruction.java:1500)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction
(ProgramBlock.java:309)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions
(ProgramBlock.java:227)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute
(ProgramBlock.java:169)
>> > > > > > at
>> > > > > >
>> > > >
>> >
>> org.apache.sysml.runtime.controlprogram.Program.execute
(Program.java:146)
>> > > > > > at org.apache.sysml.api.DMLScript.execute
(DMLScript.java:676)
>> > > > > > at
>> > > org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
>> > > > > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
>> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
>> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
>> > > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Ethan Xu <et...@gmail.com>.
Seems it's a problem of
String taskType = (conf.getBoolean(JobContext.TASK_ISMAP, true)) ? "m" : "r"
;
on line 137 of
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter
conf.getBoolean(JobContext.TASK_ISMAP, true) returns 'true' (perhaps using
provided default value) even in reducers
causing (line 139) 'charIx' to get value -1 that leads to the error
Confirmed on my test case. When error occurs in reducer, 'name' has value
'0-r-00000', but 'taskType' has value 'm'.
The following (ugly) hack fixed the problem. DML ran successfully
afterwards.
Change line 137 - 139 of
org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter
to:
String name = file.getName();
Pattern p = Pattern.compile("-[rm]-");
Matcher m = p.matcher(name);
int charIx = 0;
if (m.find()) {
charIx = m.start();
}else{
throw new RuntimeException("file name :" + name + "doesn't contain 'r'
or 'm'");
}
Ethan
On Fri, Feb 5, 2016 at 4:37 PM, Ethan Xu <et...@gmail.com> wrote:
> Thanks tried that and moved a bit further. Now a new exception (still in
> reduce phase of 'CSV-Reblock-MR'):
>
> WARN org.apache.hadoop.mapred.Child: Error running child
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1911)
> at org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.moveFileToDestination(MultipleOutputCommitter.java:140)
> at org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.moveFinalTaskOutputs(MultipleOutputCommitter.java:119)
> at org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.commitTask(MultipleOutputCommitter.java:94)
> at org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:221)
> at org.apache.hadoop.mapred.Task.commit(Task.java:1005)
> at org.apache.hadoop.mapred.Task.done(Task.java:875)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:453)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
>
>
> On Fri, Feb 5, 2016 at 4:03 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
>
>> ok that is interesting. I think the following is happening: The hadoop
>> version is >2.0, which makes SystemML switch to the 2.x configuration
>> properties. However, because MR1 is bundled into this distribution these
>> configurations do not exist which makes us fail on processing task ids.
>>
>> Workaround: Change
>> org.apache.sysml.runtime.matrix.mapred.MRConfigurationNames line 85 to
>> *"boolean* hadoopVersion2 = false".
>>
>> Regards,
>> Matthias
>>
>> [image: Inactive hide details for Ethan Xu ---02/05/2016 12:36:27
>> PM---Thank you very much. I just pulled the update, rebuilt the proje]Ethan
>> Xu ---02/05/2016 12:36:27 PM---Thank you very much. I just pulled the
>> update, rebuilt the project and reran the code.
>>
>> From: Ethan Xu <et...@gmail.com>
>> To: dev@systemml.incubator.apache.org
>> Date: 02/05/2016 12:36 PM
>> Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
>> ------------------------------
>>
>>
>>
>> Thank you very much. I just pulled the update, rebuilt the project and
>> reran the code.
>>
>> The method-not-found error was gone, and the MapReduce job was kicked off.
>> The 'Assign-RowID-MR' job finished successfully.
>> The map phase of 'CSV-Reblock-MR' job finished, but reducers threw
>> NullPointerExceptions at
>>
>> java.lang.NullPointerException
>> at
>> org.apache.sysml.runtime.matrix.mapred.ReduceBase.close(ReduceBase.java:205)
>> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:516)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>
>> The job I ran was the same as before on the same data:
>> hadoop jar <SystemML dir>/target/SystemML.jar -libjars <local
>> dir>/hadoop-lzo-0.4.15.jar -f <SystemML
>> dir>/scripts/algorithms/Univar-Stats.dml -nvargs X=<HDFS
>> dir>/original-coded.csv TYPES=<HDFS dir>/original-coded-type.csv
>> STATS=<HDFS dir>/univariate-summary.csv
>>
>> The hadoop cluster was also the same one: CDH4.2.1.
>>
>> Sorry for keep coming back with problems on a really old hadoop system.
>> Please let me know what other information is needed to diagnose the issue.
>>
>> Ethan
>>
>>
>> On Fri, Feb 5, 2016 at 1:26 PM, Deron Eriksson <de...@gmail.com>
>> wrote:
>>
>> > Hi Ethan,
>> >
>> > I believe your safest, cleanest bet is to wait for the fix from
>> Matthias.
>> > When he pushes the fix, you will see it at
>> > https://github.com/apache/incubator-systemml/commits/master. At that
>> > point,
>> > you can pull (git pull) the changes from GitHub to your machine and then
>> > build with Maven utilizing the new changes.
>> >
>> > Alternatively, it's not really recommended, but you might be able to use
>> > -libjars to reference the hadoop-commons jar, which should be in your
>> local
>> > maven repository
>> >
>> >
>> (.m2/repository/org/apache/hadoop/hadoop-common/2.4.1/hadoop-common-2.4.1.jar).
>> > However, mixing jar versions usually doesn't work very well (it can
>> lead to
>> > other problems), so waiting for the fix is best.
>> >
>> > Deron
>> >
>> >
>> > On Fri, Feb 5, 2016 at 6:47 AM, Ethan Xu <et...@gmail.com>
>> wrote:
>> >
>> > > Thank you Shirish and Deron for the suggestions. Looking forward to
>> the
>> > fix
>> > > from Matthias!
>> > >
>> > > We are using the hadoop-common shipped with CDH4.2.1, and it's in
>> > > classpath. I'm a bit hesitate to alter our hadoop configuration to
>> > include
>> > > other versions since other people are using it too.
>> > >
>> > > Not sure if/how the following naive approach affects the program
>> > behavior,
>> > > but I did try changing the scope of
>> > >
>> > > <groupId>org.apache.hadoop</groupId>
>> > > <artifactId>hadoop-common</artifactId>
>> > > <version>${hadoop.version}</version>
>> > >
>> > > in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the jar
>> > > (21MB), and it threw the same error.
>> > >
>> > > By the way this is in pom.xml line 65 - 72:
>> > > <properties>
>> > > <hadoop.version>2.4.1</hadoop.version>
>> > > <antlr.version>4.3</antlr.version>
>> > > <spark.version>1.4.1</spark.version>
>> > >
>> > > <!-- OS-specific JVM arguments for running integration
>> > > tests -->
>> > > <integrationTestExtraJVMArgs />
>> > > </properties>
>> > >
>> > > Am I supposed to modify the hadoop.version before build?
>> > >
>> > > Thanks again,
>> > >
>> > > Ethan
>> > >
>> > >
>> > >
>> > > On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson <
>> deroneriksson@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Matthias,
>> > > >
>> > > > Glad to hear the fix is simple. Mixing jar versions sometimes is not
>> > very
>> > > > fun.
>> > > >
>> > > > Deron
>> > > >
>> > > >
>> > > > On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com>
>> > > wrote:
>> > > >
>> > > > > well, let's not mix different hadoop versions in the class path or
>> > > > > client/server. If I'm not mistaken, cdh 4.x always shipped with MR
>> > v1.
>> > > > It's
>> > > > > a trivial fix for us and will be in the repo tomorrow morning
>> anyway.
>> > > > > Thanks for catching this issue Ethan.
>> > > > >
>> > > > > Regards,
>> > > > > Matthias
>> > > > >
>> > > > > [image: Inactive hide details for Deron Eriksson ---02/04/2016
>> > 11:04:38
>> > > > > PM---Hi Ethan, Just FYI, I looked at
>> hadoop-common-2.0.0-cdh4.2]Deron
>> > > > > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked
>> at
>> > > > > hadoop-common-2.0.0-cdh4.2.1.jar (
>> > > > >
>> > > > > From: Deron Eriksson <de...@gmail.com>
>> > > > > To: dev@systemml.incubator.apache.org
>> > > > > Date: 02/04/2016 11:04 PM
>> > > > > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
>> > > > > ------------------------------
>> > > > >
>> > > > >
>> > > > >
>> > > > > Hi Ethan,
>> > > > >
>> > > > > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
>> > > > > ),
>> > > > > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
>> > > > > org.apache.hadoop.conf.Configuration class in that jar doesn't
>> appear
>> > > to
>> > > > > have a getDouble method, so using that version of hadoop-common
>> won't
>> > > > work.
>> > > > >
>> > > > > However, the hadoop-common-2.4.1.jar (
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
>> > > > > )
>> > > > >
>> > > > > does appear to have the getDouble method. It's possible that
>> adding
>> > > that
>> > > > > jar to your classpath may fix your problem, as Shirish pointed
>> out.
>> > > > >
>> > > > > It sounds like Matthias may have another fix.
>> > > > >
>> > > > > Deron
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mboehm@us.ibm.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > well, we did indeed not run on MR v1 for a while now. However, I
>> > > don't
>> > > > > > want to get that far and say we don't support it anymore. I'll
>> fix
>> > > this
>> > > > > > particular issue by tomorrow.
>> > > > > >
>> > > > > > In the next couple of weeks we should run our full performance
>> > > > testsuite
>> > > > > > (for broad coverage) over an MR v1 cluster and systematically
>> > remove
>> > > > > > unnecessary incompatibility like this instance. Any volunteers?
>> > > > > >
>> > > > > > Regards,
>> > > > > > Matthias
>> > > > > >
>> > > > > > [image: Inactive hide details for Ethan Xu ---02/04/2016
>> 05:51:28
>> > > > > > PM---Hello, I got an error when running the
>> > > > > systemML/scripts/Univar-S]Ethan
>> > > > > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when
>> running
>> > the
>> > > > > > systemML/scripts/Univar-Stats.dml script on
>> > > > > >
>> > > > > > From: Ethan Xu <et...@gmail.com>
>> > > > > > To: dev@systemml.incubator.apache.org
>> > > > > > Date: 02/04/2016 05:51 PM
>> > > > > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
>> > > > > > ------------------------------
>> > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Hello,
>> > > > > >
>> > > > > > I got an error when running the
>> systemML/scripts/Univar-Stats.dml
>> > > > script
>> > > > > on
>> > > > > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error
>> > message
>> > > > is
>> > > > > at
>> > > > > > the bottom of the email. The same script ran fine on a smaller
>> > sample
>> > > > > > (several MB) of the same data set, when MR was not invoked.
>> > > > > >
>> > > > > > The main error was java.lang.NoSuchMethodError:
>> > > > > > org.apache.hadoop.mapred.JobConf.getDouble()
>> > > > > > Digging deeper, it looks like the CDH4.2.1 version of MR indeed
>> > > didn't
>> > > > > have
>> > > > > > the JobConf.getDouble() method.
>> > > > > >
>> > > > > > The hadoop-core jar of CDH4.2.1 can be found here:
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>> > > > >
>> > > > > >
>> > > > > > The calling line of SystemML is line 1194 of
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>> > > > > >
>> > > > > > I was wondering, if the finding is accurate, is there a
>> potential
>> > > fix,
>> > > > or
>> > > > > > does this mean the current version of SystemML is not compatible
>> > with
>> > > > > > CDH4.2.1?
>> > > > > >
>> > > > > > Thank you,
>> > > > > >
>> > > > > > Ethan
>> > > > > >
>> > > > > >
>> > > > > > hadoop jar $sysDir/target/SystemML.jar -f
>> > > > > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
>> > > > > > X=$baseDirHDFS/original-coded.csv
>> > > > > > TYPES=$baseDirHDFS/original-coded-type.csv
>> > > > > > STATS=$baseDirHDFS/univariate-summary.csv
>> > > > > >
>> > > > > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016
>> > > 20:35:03
>> > > > > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
>> > > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML
>> config
>> > > file
>> > > > > > (./SystemML-config.xml) found
>> > > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
>> > > > > DMLConfig
>> > > > > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable
>> > > multi-threaded
>> > > > > > text read for 'text' and 'csv' due to thread contention on JRE <
>> > 1.8
>> > > > > > (java.version=1.7.0_71).
>> > > > > > SLF4J: Class path contains multiple SLF4J bindings.
>> > > > > > SLF4J: Found binding in
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > > > > > SLF4J: Found binding in
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > > > > > SLF4J: Found binding in
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings
>> for
>> > an
>> > > > > > explanation.
>> > > > > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
>> > > > > > Total execution time: 0.880 sec.
>> > > > > > Number of executed MR Jobs: 0.
>> > > > > >
>> > > > > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016
>> > 20:35:07
>> > > > > > Exception in thread "main" java.lang.NoSuchMethodError:
>> > > > > > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
>> > > > > > at
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
>> > > > > > at
>> > > > >
>> org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
>> > > > > > at
>> > > > > >
>> > > >
>> >
>> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
>> > > > > > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
>> > > > > > at
>> > > org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
>> > > > > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
>> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
>> > > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Ethan Xu <et...@gmail.com>.
Thanks tried that and moved a bit further. Now a new exception (still in
reduce phase of 'CSV-Reblock-MR'):
WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1911)
at org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.moveFileToDestination(MultipleOutputCommitter.java:140)
at org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.moveFinalTaskOutputs(MultipleOutputCommitter.java:119)
at org.apache.sysml.runtime.matrix.data.MultipleOutputCommitter.commitTask(MultipleOutputCommitter.java:94)
at org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:221)
at org.apache.hadoop.mapred.Task.commit(Task.java:1005)
at org.apache.hadoop.mapred.Task.done(Task.java:875)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:453)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
On Fri, Feb 5, 2016 at 4:03 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
> ok that is interesting. I think the following is happening: The hadoop
> version is >2.0, which makes SystemML switch to the 2.x configuration
> properties. However, because MR1 is bundled into this distribution these
> configurations do not exist which makes us fail on processing task ids.
>
> Workaround: Change
> org.apache.sysml.runtime.matrix.mapred.MRConfigurationNames line 85 to
> *"boolean* hadoopVersion2 = false".
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Ethan Xu ---02/05/2016 12:36:27
> PM---Thank you very much. I just pulled the update, rebuilt the proje]Ethan
> Xu ---02/05/2016 12:36:27 PM---Thank you very much. I just pulled the
> update, rebuilt the project and reran the code.
>
> From: Ethan Xu <et...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 02/05/2016 12:36 PM
> Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> ------------------------------
>
>
>
> Thank you very much. I just pulled the update, rebuilt the project and
> reran the code.
>
> The method-not-found error was gone, and the MapReduce job was kicked off.
> The 'Assign-RowID-MR' job finished successfully.
> The map phase of 'CSV-Reblock-MR' job finished, but reducers threw
> NullPointerExceptions at
>
> java.lang.NullPointerException
> at
> org.apache.sysml.runtime.matrix.mapred.ReduceBase.close(ReduceBase.java:205)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:516)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
> The job I ran was the same as before on the same data:
> hadoop jar <SystemML dir>/target/SystemML.jar -libjars <local
> dir>/hadoop-lzo-0.4.15.jar -f <SystemML
> dir>/scripts/algorithms/Univar-Stats.dml -nvargs X=<HDFS
> dir>/original-coded.csv TYPES=<HDFS dir>/original-coded-type.csv
> STATS=<HDFS dir>/univariate-summary.csv
>
> The hadoop cluster was also the same one: CDH4.2.1.
>
> Sorry for keep coming back with problems on a really old hadoop system.
> Please let me know what other information is needed to diagnose the issue.
>
> Ethan
>
>
> On Fri, Feb 5, 2016 at 1:26 PM, Deron Eriksson <de...@gmail.com>
> wrote:
>
> > Hi Ethan,
> >
> > I believe your safest, cleanest bet is to wait for the fix from Matthias.
> > When he pushes the fix, you will see it at
> > https://github.com/apache/incubator-systemml/commits/master. At that
> > point,
> > you can pull (git pull) the changes from GitHub to your machine and then
> > build with Maven utilizing the new changes.
> >
> > Alternatively, it's not really recommended, but you might be able to use
> > -libjars to reference the hadoop-commons jar, which should be in your
> local
> > maven repository
> >
> >
> (.m2/repository/org/apache/hadoop/hadoop-common/2.4.1/hadoop-common-2.4.1.jar).
> > However, mixing jar versions usually doesn't work very well (it can lead
> to
> > other problems), so waiting for the fix is best.
> >
> > Deron
> >
> >
> > On Fri, Feb 5, 2016 at 6:47 AM, Ethan Xu <et...@gmail.com>
> wrote:
> >
> > > Thank you Shirish and Deron for the suggestions. Looking forward to the
> > fix
> > > from Matthias!
> > >
> > > We are using the hadoop-common shipped with CDH4.2.1, and it's in
> > > classpath. I'm a bit hesitate to alter our hadoop configuration to
> > include
> > > other versions since other people are using it too.
> > >
> > > Not sure if/how the following naive approach affects the program
> > behavior,
> > > but I did try changing the scope of
> > >
> > > <groupId>org.apache.hadoop</groupId>
> > > <artifactId>hadoop-common</artifactId>
> > > <version>${hadoop.version}</version>
> > >
> > > in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the jar
> > > (21MB), and it threw the same error.
> > >
> > > By the way this is in pom.xml line 65 - 72:
> > > <properties>
> > > <hadoop.version>2.4.1</hadoop.version>
> > > <antlr.version>4.3</antlr.version>
> > > <spark.version>1.4.1</spark.version>
> > >
> > > <!-- OS-specific JVM arguments for running integration
> > > tests -->
> > > <integrationTestExtraJVMArgs />
> > > </properties>
> > >
> > > Am I supposed to modify the hadoop.version before build?
> > >
> > > Thanks again,
> > >
> > > Ethan
> > >
> > >
> > >
> > > On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson <
> deroneriksson@gmail.com>
> > > wrote:
> > >
> > > > Hi Matthias,
> > > >
> > > > Glad to hear the fix is simple. Mixing jar versions sometimes is not
> > very
> > > > fun.
> > > >
> > > > Deron
> > > >
> > > >
> > > > On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com>
> > > wrote:
> > > >
> > > > > well, let's not mix different hadoop versions in the class path or
> > > > > client/server. If I'm not mistaken, cdh 4.x always shipped with MR
> > v1.
> > > > It's
> > > > > a trivial fix for us and will be in the repo tomorrow morning
> anyway.
> > > > > Thanks for catching this issue Ethan.
> > > > >
> > > > > Regards,
> > > > > Matthias
> > > > >
> > > > > [image: Inactive hide details for Deron Eriksson ---02/04/2016
> > 11:04:38
> > > > > PM---Hi Ethan, Just FYI, I looked at
> hadoop-common-2.0.0-cdh4.2]Deron
> > > > > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked
> at
> > > > > hadoop-common-2.0.0-cdh4.2.1.jar (
> > > > >
> > > > > From: Deron Eriksson <de...@gmail.com>
> > > > > To: dev@systemml.incubator.apache.org
> > > > > Date: 02/04/2016 11:04 PM
> > > > > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> > > > > ------------------------------
> > > > >
> > > > >
> > > > >
> > > > > Hi Ethan,
> > > > >
> > > > > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
> > > > >
> > > > >
> > > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
> > > > > ),
> > > > > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
> > > > > org.apache.hadoop.conf.Configuration class in that jar doesn't
> appear
> > > to
> > > > > have a getDouble method, so using that version of hadoop-common
> won't
> > > > work.
> > > > >
> > > > > However, the hadoop-common-2.4.1.jar (
> > > > >
> > > > >
> > > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
> > > > > )
> > > > >
> > > > > does appear to have the getDouble method. It's possible that adding
> > > that
> > > > > jar to your classpath may fix your problem, as Shirish pointed out.
> > > > >
> > > > > It sounds like Matthias may have another fix.
> > > > >
> > > > > Deron
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com>
> > > > wrote:
> > > > >
> > > > > > well, we did indeed not run on MR v1 for a while now. However, I
> > > don't
> > > > > > want to get that far and say we don't support it anymore. I'll
> fix
> > > this
> > > > > > particular issue by tomorrow.
> > > > > >
> > > > > > In the next couple of weeks we should run our full performance
> > > > testsuite
> > > > > > (for broad coverage) over an MR v1 cluster and systematically
> > remove
> > > > > > unnecessary incompatibility like this instance. Any volunteers?
> > > > > >
> > > > > > Regards,
> > > > > > Matthias
> > > > > >
> > > > > > [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> > > > > > PM---Hello, I got an error when running the
> > > > > systemML/scripts/Univar-S]Ethan
> > > > > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running
> > the
> > > > > > systemML/scripts/Univar-Stats.dml script on
> > > > > >
> > > > > > From: Ethan Xu <et...@gmail.com>
> > > > > > To: dev@systemml.incubator.apache.org
> > > > > > Date: 02/04/2016 05:51 PM
> > > > > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
> > > > > > ------------------------------
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I got an error when running the systemML/scripts/Univar-Stats.dml
> > > > script
> > > > > on
> > > > > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error
> > message
> > > > is
> > > > > at
> > > > > > the bottom of the email. The same script ran fine on a smaller
> > sample
> > > > > > (several MB) of the same data set, when MR was not invoked.
> > > > > >
> > > > > > The main error was java.lang.NoSuchMethodError:
> > > > > > org.apache.hadoop.mapred.JobConf.getDouble()
> > > > > > Digging deeper, it looks like the CDH4.2.1 version of MR indeed
> > > didn't
> > > > > have
> > > > > > the JobConf.getDouble() method.
> > > > > >
> > > > > > The hadoop-core jar of CDH4.2.1 can be found here:
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
> > > > >
> > > > > >
> > > > > > The calling line of SystemML is line 1194 of
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
> > > > > >
> > > > > > I was wondering, if the finding is accurate, is there a potential
> > > fix,
> > > > or
> > > > > > does this mean the current version of SystemML is not compatible
> > with
> > > > > > CDH4.2.1?
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Ethan
> > > > > >
> > > > > >
> > > > > > hadoop jar $sysDir/target/SystemML.jar -f
> > > > > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> > > > > > X=$baseDirHDFS/original-coded.csv
> > > > > > TYPES=$baseDirHDFS/original-coded-type.csv
> > > > > > STATS=$baseDirHDFS/univariate-summary.csv
> > > > > >
> > > > > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016
> > > 20:35:03
> > > > > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> > > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config
> > > file
> > > > > > (./SystemML-config.xml) found
> > > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
> > > > > DMLConfig
> > > > > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable
> > > multi-threaded
> > > > > > text read for 'text' and 'csv' due to thread contention on JRE <
> > 1.8
> > > > > > (java.version=1.7.0_71).
> > > > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > > > SLF4J: Found binding in
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > > SLF4J: Found binding in
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > > SLF4J: Found binding in
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> > an
> > > > > > explanation.
> > > > > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> > > > > > Total execution time: 0.880 sec.
> > > > > > Number of executed MR Jobs: 0.
> > > > > >
> > > > > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016
> > 20:35:07
> > > > > > Exception in thread "main" java.lang.NoSuchMethodError:
> > > > > > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> > > > > > at
> > > > >
> org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> > > > > > at
> > > > > >
> > > >
> > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> > > > > > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> > > > > > at
> > > org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> > > > > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Matthias Boehm <mb...@us.ibm.com>.
ok that is interesting. I think the following is happening: The hadoop
version is >2.0, which makes SystemML switch to the 2.x configuration
properties. However, because MR1 is bundled into this distribution these
configurations do not exist which makes us fail on processing task ids.
Workaround: Change
org.apache.sysml.runtime.matrix.mapred.MRConfigurationNames line 85 to
"boolean hadoopVersion2 = false".
Regards,
Matthias
From: Ethan Xu <et...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 02/05/2016 12:36 PM
Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
Thank you very much. I just pulled the update, rebuilt the project and
reran the code.
The method-not-found error was gone, and the MapReduce job was kicked off.
The 'Assign-RowID-MR' job finished successfully.
The map phase of 'CSV-Reblock-MR' job finished, but reducers threw
NullPointerExceptions at
java.lang.NullPointerException
at org.apache.sysml.runtime.matrix.mapred.ReduceBase.close
(ReduceBase.java:205)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer
(ReduceTask.java:516)
at org.apache.hadoop.mapred.ReduceTask.run
(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs
(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
The job I ran was the same as before on the same data:
hadoop jar <SystemML dir>/target/SystemML.jar -libjars <local
dir>/hadoop-lzo-0.4.15.jar -f <SystemML
dir>/scripts/algorithms/Univar-Stats.dml -nvargs X=<HDFS
dir>/original-coded.csv TYPES=<HDFS dir>/original-coded-type.csv
STATS=<HDFS dir>/univariate-summary.csv
The hadoop cluster was also the same one: CDH4.2.1.
Sorry for keep coming back with problems on a really old hadoop system.
Please let me know what other information is needed to diagnose the issue.
Ethan
On Fri, Feb 5, 2016 at 1:26 PM, Deron Eriksson <de...@gmail.com>
wrote:
> Hi Ethan,
>
> I believe your safest, cleanest bet is to wait for the fix from Matthias.
> When he pushes the fix, you will see it at
> https://github.com/apache/incubator-systemml/commits/master. At that
> point,
> you can pull (git pull) the changes from GitHub to your machine and then
> build with Maven utilizing the new changes.
>
> Alternatively, it's not really recommended, but you might be able to use
> -libjars to reference the hadoop-commons jar, which should be in your
local
> maven repository
>
>
(.m2/repository/org/apache/hadoop/hadoop-common/2.4.1/hadoop-common-2.4.1.jar).
> However, mixing jar versions usually doesn't work very well (it can lead
to
> other problems), so waiting for the fix is best.
>
> Deron
>
>
> On Fri, Feb 5, 2016 at 6:47 AM, Ethan Xu <et...@gmail.com> wrote:
>
> > Thank you Shirish and Deron for the suggestions. Looking forward to the
> fix
> > from Matthias!
> >
> > We are using the hadoop-common shipped with CDH4.2.1, and it's in
> > classpath. I'm a bit hesitate to alter our hadoop configuration to
> include
> > other versions since other people are using it too.
> >
> > Not sure if/how the following naive approach affects the program
> behavior,
> > but I did try changing the scope of
> >
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-common</artifactId>
> > <version>${hadoop.version}</version>
> >
> > in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the jar
> > (21MB), and it threw the same error.
> >
> > By the way this is in pom.xml line 65 - 72:
> > <properties>
> > <hadoop.version>2.4.1</hadoop.version>
> > <antlr.version>4.3</antlr.version>
> > <spark.version>1.4.1</spark.version>
> >
> > <!-- OS-specific JVM arguments for running integration
>
> tests -->
> > <integrationTestExtraJVMArgs />
> > </properties>
> >
> > Am I supposed to modify the hadoop.version before build?
> >
> > Thanks again,
> >
> > Ethan
> >
> >
> >
> > On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson
<de...@gmail.com>
> > wrote:
> >
> > > Hi Matthias,
> > >
> > > Glad to hear the fix is simple. Mixing jar versions sometimes is not
> very
> > > fun.
> > >
> > > Deron
> > >
> > >
> > > On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com>
> > wrote:
> > >
> > > > well, let's not mix different hadoop versions in the class path or
> > > > client/server. If I'm not mistaken, cdh 4.x always shipped with MR
> v1.
> > > It's
> > > > a trivial fix for us and will be in the repo tomorrow morning
anyway.
> > > > Thanks for catching this issue Ethan.
> > > >
> > > > Regards,
> > > > Matthias
> > > >
> > > > [image: Inactive hide details for Deron Eriksson ---02/04/2016
> 11:04:38
> > > > PM---Hi Ethan, Just FYI, I looked at
hadoop-common-2.0.0-cdh4.2]Deron
> > > > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked
at
> > > > hadoop-common-2.0.0-cdh4.2.1.jar (
> > > >
> > > > From: Deron Eriksson <de...@gmail.com>
> > > > To: dev@systemml.incubator.apache.org
> > > > Date: 02/04/2016 11:04 PM
> > > > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> > > > ------------------------------
> > > >
> > > >
> > > >
> > > > Hi Ethan,
> > > >
> > > > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
> > > >
> > > >
> > >
> >
>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
> > > > ),
> > > > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
> > > > org.apache.hadoop.conf.Configuration class in that jar doesn't
appear
> > to
> > > > have a getDouble method, so using that version of hadoop-common
won't
> > > work.
> > > >
> > > > However, the hadoop-common-2.4.1.jar (
> > > >
> > > >
> > >
> >
>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
> > > > )
> > > >
> > > > does appear to have the getDouble method. It's possible that adding
> > that
> > > > jar to your classpath may fix your problem, as Shirish pointed out.
> > > >
> > > > It sounds like Matthias may have another fix.
> > > >
> > > > Deron
> > > >
> > > >
> > > >
> > > > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com>
> > > wrote:
> > > >
> > > > > well, we did indeed not run on MR v1 for a while now. However, I
> > don't
> > > > > want to get that far and say we don't support it anymore. I'll
fix
> > this
> > > > > particular issue by tomorrow.
> > > > >
> > > > > In the next couple of weeks we should run our full performance
> > > testsuite
> > > > > (for broad coverage) over an MR v1 cluster and systematically
> remove
> > > > > unnecessary incompatibility like this instance. Any volunteers?
> > > > >
> > > > > Regards,
> > > > > Matthias
> > > > >
> > > > > [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> > > > > PM---Hello, I got an error when running the
> > > > systemML/scripts/Univar-S]Ethan
> > > > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running
> the
> > > > > systemML/scripts/Univar-Stats.dml script on
> > > > >
> > > > > From: Ethan Xu <et...@gmail.com>
> > > > > To: dev@systemml.incubator.apache.org
> > > > > Date: 02/04/2016 05:51 PM
> > > > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
> > > > > ------------------------------
> > > >
> > > > >
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > I got an error when running the systemML/scripts/Univar-Stats.dml
> > > script
> > > > on
> > > > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error
> message
> > > is
> > > > at
> > > > > the bottom of the email. The same script ran fine on a smaller
> sample
> > > > > (several MB) of the same data set, when MR was not invoked.
> > > > >
> > > > > The main error was java.lang.NoSuchMethodError:
> > > > > org.apache.hadoop.mapred.JobConf.getDouble()
> > > > > Digging deeper, it looks like the CDH4.2.1 version of MR indeed
> > didn't
> > > > have
> > > > > the JobConf.getDouble() method.
> > > > >
> > > > > The hadoop-core jar of CDH4.2.1 can be found here:
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
> > > >
> > > > >
> > > > > The calling line of SystemML is line 1194 of
> > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
> > > > >
> > > > > I was wondering, if the finding is accurate, is there a potential
> > fix,
> > > or
> > > > > does this mean the current version of SystemML is not compatible
> with
> > > > > CDH4.2.1?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Ethan
> > > > >
> > > > >
> > > > > hadoop jar $sysDir/target/SystemML.jar -f
> > > > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> > > > > X=$baseDirHDFS/original-coded.csv
> > > > > TYPES=$baseDirHDFS/original-coded-type.csv
> > > > > STATS=$baseDirHDFS/univariate-summary.csv
> > > > >
> > > > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016
> > 20:35:03
> > > > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config
> > file
> > > > > (./SystemML-config.xml) found
> > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
> > > > DMLConfig
> > > > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable
> > multi-threaded
> > > > > text read for 'text' and 'csv' due to thread contention on JRE <
> 1.8
> > > > > (java.version=1.7.0_71).
> > > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > > SLF4J: Found binding in
> > > > >
> > > > >
> > > >
> > >
> >
>
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > SLF4J: Found binding in
> > > > >
> > > > >
> > > >
> > >
> >
>
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > SLF4J: Found binding in
> > > > >
> > > > >
> > > >
> > >
> >
>
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an
> > > > > explanation.
> > > > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> > > > > Total execution time: 0.880 sec.
> > > > > Number of executed MR Jobs: 0.
> > > > >
> > > > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016
> 20:35:07
> > > > > Exception in thread "main" java.lang.NoSuchMethodError:
> > > > > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1195)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1129)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:307)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:289)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob
(CSVReblockMR.java:275)
> > > > > at
> > > > org.apache.sysml.lops.runtime.RunMRJobs.submitJob
(RunMRJobs.java:257)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob
(RunMRJobs.java:143)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction
(MRJobInstruction.java:1500)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction
(ProgramBlock.java:309)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions
(ProgramBlock.java:227)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute
(ProgramBlock.java:169)
> > > > > at
> > > > >
> > >
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> > > > > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> > > > > at
> > org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> > > > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Ethan Xu <et...@gmail.com>.
Thank you very much. I just pulled the update, rebuilt the project and
reran the code.
The method-not-found error was gone, and the MapReduce job was kicked off.
The 'Assign-RowID-MR' job finished successfully.
The map phase of 'CSV-Reblock-MR' job finished, but reducers threw
NullPointerExceptions at
java.lang.NullPointerException
at org.apache.sysml.runtime.matrix.mapred.ReduceBase.close(ReduceBase.java:205)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:516)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
The job I ran was the same as before on the same data:
hadoop jar <SystemML dir>/target/SystemML.jar -libjars <local
dir>/hadoop-lzo-0.4.15.jar -f <SystemML
dir>/scripts/algorithms/Univar-Stats.dml -nvargs X=<HDFS
dir>/original-coded.csv TYPES=<HDFS dir>/original-coded-type.csv
STATS=<HDFS dir>/univariate-summary.csv
The hadoop cluster was also the same one: CDH4.2.1.
Sorry for keep coming back with problems on a really old hadoop system.
Please let me know what other information is needed to diagnose the issue.
Ethan
On Fri, Feb 5, 2016 at 1:26 PM, Deron Eriksson <de...@gmail.com>
wrote:
> Hi Ethan,
>
> I believe your safest, cleanest bet is to wait for the fix from Matthias.
> When he pushes the fix, you will see it at
> https://github.com/apache/incubator-systemml/commits/master. At that
> point,
> you can pull (git pull) the changes from GitHub to your machine and then
> build with Maven utilizing the new changes.
>
> Alternatively, it's not really recommended, but you might be able to use
> -libjars to reference the hadoop-commons jar, which should be in your local
> maven repository
>
> (.m2/repository/org/apache/hadoop/hadoop-common/2.4.1/hadoop-common-2.4.1.jar).
> However, mixing jar versions usually doesn't work very well (it can lead to
> other problems), so waiting for the fix is best.
>
> Deron
>
>
> On Fri, Feb 5, 2016 at 6:47 AM, Ethan Xu <et...@gmail.com> wrote:
>
> > Thank you Shirish and Deron for the suggestions. Looking forward to the
> fix
> > from Matthias!
> >
> > We are using the hadoop-common shipped with CDH4.2.1, and it's in
> > classpath. I'm a bit hesitate to alter our hadoop configuration to
> include
> > other versions since other people are using it too.
> >
> > Not sure if/how the following naive approach affects the program
> behavior,
> > but I did try changing the scope of
> >
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-common</artifactId>
> > <version>${hadoop.version}</version>
> >
> > in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the jar
> > (21MB), and it threw the same error.
> >
> > By the way this is in pom.xml line 65 - 72:
> > <properties>
> > <hadoop.version>2.4.1</hadoop.version>
> > <antlr.version>4.3</antlr.version>
> > <spark.version>1.4.1</spark.version>
> >
> > <!-- OS-specific JVM arguments for running integration
> > tests -->
> > <integrationTestExtraJVMArgs />
> > </properties>
> >
> > Am I supposed to modify the hadoop.version before build?
> >
> > Thanks again,
> >
> > Ethan
> >
> >
> >
> > On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson <de...@gmail.com>
> > wrote:
> >
> > > Hi Matthias,
> > >
> > > Glad to hear the fix is simple. Mixing jar versions sometimes is not
> very
> > > fun.
> > >
> > > Deron
> > >
> > >
> > > On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com>
> > wrote:
> > >
> > > > well, let's not mix different hadoop versions in the class path or
> > > > client/server. If I'm not mistaken, cdh 4.x always shipped with MR
> v1.
> > > It's
> > > > a trivial fix for us and will be in the repo tomorrow morning anyway.
> > > > Thanks for catching this issue Ethan.
> > > >
> > > > Regards,
> > > > Matthias
> > > >
> > > > [image: Inactive hide details for Deron Eriksson ---02/04/2016
> 11:04:38
> > > > PM---Hi Ethan, Just FYI, I looked at hadoop-common-2.0.0-cdh4.2]Deron
> > > > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked at
> > > > hadoop-common-2.0.0-cdh4.2.1.jar (
> > > >
> > > > From: Deron Eriksson <de...@gmail.com>
> > > > To: dev@systemml.incubator.apache.org
> > > > Date: 02/04/2016 11:04 PM
> > > > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> > > > ------------------------------
> > > >
> > > >
> > > >
> > > > Hi Ethan,
> > > >
> > > > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
> > > >
> > > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
> > > > ),
> > > > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
> > > > org.apache.hadoop.conf.Configuration class in that jar doesn't appear
> > to
> > > > have a getDouble method, so using that version of hadoop-common won't
> > > work.
> > > >
> > > > However, the hadoop-common-2.4.1.jar (
> > > >
> > > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
> > > > )
> > > >
> > > > does appear to have the getDouble method. It's possible that adding
> > that
> > > > jar to your classpath may fix your problem, as Shirish pointed out.
> > > >
> > > > It sounds like Matthias may have another fix.
> > > >
> > > > Deron
> > > >
> > > >
> > > >
> > > > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com>
> > > wrote:
> > > >
> > > > > well, we did indeed not run on MR v1 for a while now. However, I
> > don't
> > > > > want to get that far and say we don't support it anymore. I'll fix
> > this
> > > > > particular issue by tomorrow.
> > > > >
> > > > > In the next couple of weeks we should run our full performance
> > > testsuite
> > > > > (for broad coverage) over an MR v1 cluster and systematically
> remove
> > > > > unnecessary incompatibility like this instance. Any volunteers?
> > > > >
> > > > > Regards,
> > > > > Matthias
> > > > >
> > > > > [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> > > > > PM---Hello, I got an error when running the
> > > > systemML/scripts/Univar-S]Ethan
> > > > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running
> the
> > > > > systemML/scripts/Univar-Stats.dml script on
> > > > >
> > > > > From: Ethan Xu <et...@gmail.com>
> > > > > To: dev@systemml.incubator.apache.org
> > > > > Date: 02/04/2016 05:51 PM
> > > > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
> > > > > ------------------------------
> > > >
> > > > >
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > I got an error when running the systemML/scripts/Univar-Stats.dml
> > > script
> > > > on
> > > > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error
> message
> > > is
> > > > at
> > > > > the bottom of the email. The same script ran fine on a smaller
> sample
> > > > > (several MB) of the same data set, when MR was not invoked.
> > > > >
> > > > > The main error was java.lang.NoSuchMethodError:
> > > > > org.apache.hadoop.mapred.JobConf.getDouble()
> > > > > Digging deeper, it looks like the CDH4.2.1 version of MR indeed
> > didn't
> > > > have
> > > > > the JobConf.getDouble() method.
> > > > >
> > > > > The hadoop-core jar of CDH4.2.1 can be found here:
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
> > > >
> > > > >
> > > > > The calling line of SystemML is line 1194 of
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
> > > > >
> > > > > I was wondering, if the finding is accurate, is there a potential
> > fix,
> > > or
> > > > > does this mean the current version of SystemML is not compatible
> with
> > > > > CDH4.2.1?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Ethan
> > > > >
> > > > >
> > > > > hadoop jar $sysDir/target/SystemML.jar -f
> > > > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> > > > > X=$baseDirHDFS/original-coded.csv
> > > > > TYPES=$baseDirHDFS/original-coded-type.csv
> > > > > STATS=$baseDirHDFS/univariate-summary.csv
> > > > >
> > > > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016
> > 20:35:03
> > > > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config
> > file
> > > > > (./SystemML-config.xml) found
> > > > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
> > > > DMLConfig
> > > > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable
> > multi-threaded
> > > > > text read for 'text' and 'csv' due to thread contention on JRE <
> 1.8
> > > > > (java.version=1.7.0_71).
> > > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > > SLF4J: Found binding in
> > > > >
> > > > >
> > > >
> > >
> >
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > SLF4J: Found binding in
> > > > >
> > > > >
> > > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > SLF4J: Found binding in
> > > > >
> > > > >
> > > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an
> > > > > explanation.
> > > > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> > > > > Total execution time: 0.880 sec.
> > > > > Number of executed MR Jobs: 0.
> > > > >
> > > > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016
> 20:35:07
> > > > > Exception in thread "main" java.lang.NoSuchMethodError:
> > > > > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> > > > > at
> > > > org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> > > > > at
> > > > >
> > >
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> > > > > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> > > > > at
> > org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> > > > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Deron Eriksson <de...@gmail.com>.
Hi Ethan,
I believe your safest, cleanest bet is to wait for the fix from Matthias.
When he pushes the fix, you will see it at
https://github.com/apache/incubator-systemml/commits/master. At that point,
you can pull (git pull) the changes from GitHub to your machine and then
build with Maven utilizing the new changes.
Alternatively, it's not really recommended, but you might be able to use
-libjars to reference the hadoop-commons jar, which should be in your local
maven repository
(.m2/repository/org/apache/hadoop/hadoop-common/2.4.1/hadoop-common-2.4.1.jar).
However, mixing jar versions usually doesn't work very well (it can lead to
other problems), so waiting for the fix is best.
Deron
On Fri, Feb 5, 2016 at 6:47 AM, Ethan Xu <et...@gmail.com> wrote:
> Thank you Shirish and Deron for the suggestions. Looking forward to the fix
> from Matthias!
>
> We are using the hadoop-common shipped with CDH4.2.1, and it's in
> classpath. I'm a bit hesitate to alter our hadoop configuration to include
> other versions since other people are using it too.
>
> Not sure if/how the following naive approach affects the program behavior,
> but I did try changing the scope of
>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>${hadoop.version}</version>
>
> in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the jar
> (21MB), and it threw the same error.
>
> By the way this is in pom.xml line 65 - 72:
> <properties>
> <hadoop.version>2.4.1</hadoop.version>
> <antlr.version>4.3</antlr.version>
> <spark.version>1.4.1</spark.version>
>
> <!-- OS-specific JVM arguments for running integration
> tests -->
> <integrationTestExtraJVMArgs />
> </properties>
>
> Am I supposed to modify the hadoop.version before build?
>
> Thanks again,
>
> Ethan
>
>
>
> On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson <de...@gmail.com>
> wrote:
>
> > Hi Matthias,
> >
> > Glad to hear the fix is simple. Mixing jar versions sometimes is not very
> > fun.
> >
> > Deron
> >
> >
> > On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com>
> wrote:
> >
> > > well, let's not mix different hadoop versions in the class path or
> > > client/server. If I'm not mistaken, cdh 4.x always shipped with MR v1.
> > It's
> > > a trivial fix for us and will be in the repo tomorrow morning anyway.
> > > Thanks for catching this issue Ethan.
> > >
> > > Regards,
> > > Matthias
> > >
> > > [image: Inactive hide details for Deron Eriksson ---02/04/2016 11:04:38
> > > PM---Hi Ethan, Just FYI, I looked at hadoop-common-2.0.0-cdh4.2]Deron
> > > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked at
> > > hadoop-common-2.0.0-cdh4.2.1.jar (
> > >
> > > From: Deron Eriksson <de...@gmail.com>
> > > To: dev@systemml.incubator.apache.org
> > > Date: 02/04/2016 11:04 PM
> > > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> > > ------------------------------
> > >
> > >
> > >
> > > Hi Ethan,
> > >
> > > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
> > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
> > > ),
> > > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
> > > org.apache.hadoop.conf.Configuration class in that jar doesn't appear
> to
> > > have a getDouble method, so using that version of hadoop-common won't
> > work.
> > >
> > > However, the hadoop-common-2.4.1.jar (
> > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
> > > )
> > >
> > > does appear to have the getDouble method. It's possible that adding
> that
> > > jar to your classpath may fix your problem, as Shirish pointed out.
> > >
> > > It sounds like Matthias may have another fix.
> > >
> > > Deron
> > >
> > >
> > >
> > > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com>
> > wrote:
> > >
> > > > well, we did indeed not run on MR v1 for a while now. However, I
> don't
> > > > want to get that far and say we don't support it anymore. I'll fix
> this
> > > > particular issue by tomorrow.
> > > >
> > > > In the next couple of weeks we should run our full performance
> > testsuite
> > > > (for broad coverage) over an MR v1 cluster and systematically remove
> > > > unnecessary incompatibility like this instance. Any volunteers?
> > > >
> > > > Regards,
> > > > Matthias
> > > >
> > > > [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> > > > PM---Hello, I got an error when running the
> > > systemML/scripts/Univar-S]Ethan
> > > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running the
> > > > systemML/scripts/Univar-Stats.dml script on
> > > >
> > > > From: Ethan Xu <et...@gmail.com>
> > > > To: dev@systemml.incubator.apache.org
> > > > Date: 02/04/2016 05:51 PM
> > > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
> > > > ------------------------------
> > >
> > > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > I got an error when running the systemML/scripts/Univar-Stats.dml
> > script
> > > on
> > > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message
> > is
> > > at
> > > > the bottom of the email. The same script ran fine on a smaller sample
> > > > (several MB) of the same data set, when MR was not invoked.
> > > >
> > > > The main error was java.lang.NoSuchMethodError:
> > > > org.apache.hadoop.mapred.JobConf.getDouble()
> > > > Digging deeper, it looks like the CDH4.2.1 version of MR indeed
> didn't
> > > have
> > > > the JobConf.getDouble() method.
> > > >
> > > > The hadoop-core jar of CDH4.2.1 can be found here:
> > > >
> > > >
> > >
> > >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
> > >
> > > >
> > > > The calling line of SystemML is line 1194 of
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
> > > >
> > > > I was wondering, if the finding is accurate, is there a potential
> fix,
> > or
> > > > does this mean the current version of SystemML is not compatible with
> > > > CDH4.2.1?
> > > >
> > > > Thank you,
> > > >
> > > > Ethan
> > > >
> > > >
> > > > hadoop jar $sysDir/target/SystemML.jar -f
> > > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> > > > X=$baseDirHDFS/original-coded.csv
> > > > TYPES=$baseDirHDFS/original-coded-type.csv
> > > > STATS=$baseDirHDFS/univariate-summary.csv
> > > >
> > > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016
> 20:35:03
> > > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> > > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config
> file
> > > > (./SystemML-config.xml) found
> > > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
> > > DMLConfig
> > > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable
> multi-threaded
> > > > text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> > > > (java.version=1.7.0_71).
> > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > SLF4J: Found binding in
> > > >
> > > >
> > >
> >
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > SLF4J: Found binding in
> > > >
> > > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > SLF4J: Found binding in
> > > >
> > > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > > > explanation.
> > > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> > > > Total execution time: 0.880 sec.
> > > > Number of executed MR Jobs: 0.
> > > >
> > > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> > > > Exception in thread "main" java.lang.NoSuchMethodError:
> > > > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> > > > at
> > > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> > > > at
> > > org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> > > > at
> > > >
> > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> > > > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> > > > at
> org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> > > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
>
>
> --
> Yifan "Ethan" Xu, PhD
>
> Data Scientist / Statistician
> Explorys, IBM Watson Health
>
> Adjunct Faculty
> Department of Epidemiology and Biostatistics
> Case Western Reserve University
>
> --------------
> Email: ethan.yifanxu@gmail.com
> Phone: (607) 760-6817
> --------------
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Ethan Xu <et...@gmail.com>.
Thank you Shirish and Deron for the suggestions. Looking forward to the fix
from Matthias!
We are using the hadoop-common shipped with CDH4.2.1, and it's in
classpath. I'm a bit hesitate to alter our hadoop configuration to include
other versions since other people are using it too.
Not sure if/how the following naive approach affects the program behavior,
but I did try changing the scope of
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
in SystemML's pom.xml from 'provided' to 'compile' and rebuilt the jar
(21MB), and it threw the same error.
By the way this is in pom.xml line 65 - 72:
<properties>
<hadoop.version>2.4.1</hadoop.version>
<antlr.version>4.3</antlr.version>
<spark.version>1.4.1</spark.version>
<!-- OS-specific JVM arguments for running integration
tests -->
<integrationTestExtraJVMArgs />
</properties>
Am I supposed to modify the hadoop.version before build?
Thanks again,
Ethan
On Fri, Feb 5, 2016 at 2:29 AM, Deron Eriksson <de...@gmail.com>
wrote:
> Hi Matthias,
>
> Glad to hear the fix is simple. Mixing jar versions sometimes is not very
> fun.
>
> Deron
>
>
> On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
>
> > well, let's not mix different hadoop versions in the class path or
> > client/server. If I'm not mistaken, cdh 4.x always shipped with MR v1.
> It's
> > a trivial fix for us and will be in the repo tomorrow morning anyway.
> > Thanks for catching this issue Ethan.
> >
> > Regards,
> > Matthias
> >
> > [image: Inactive hide details for Deron Eriksson ---02/04/2016 11:04:38
> > PM---Hi Ethan, Just FYI, I looked at hadoop-common-2.0.0-cdh4.2]Deron
> > Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked at
> > hadoop-common-2.0.0-cdh4.2.1.jar (
> >
> > From: Deron Eriksson <de...@gmail.com>
> > To: dev@systemml.incubator.apache.org
> > Date: 02/04/2016 11:04 PM
> > Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> > ------------------------------
> >
> >
> >
> > Hi Ethan,
> >
> > Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
> >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
> > ),
> > since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
> > org.apache.hadoop.conf.Configuration class in that jar doesn't appear to
> > have a getDouble method, so using that version of hadoop-common won't
> work.
> >
> > However, the hadoop-common-2.4.1.jar (
> >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
> > )
> >
> > does appear to have the getDouble method. It's possible that adding that
> > jar to your classpath may fix your problem, as Shirish pointed out.
> >
> > It sounds like Matthias may have another fix.
> >
> > Deron
> >
> >
> >
> > On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com>
> wrote:
> >
> > > well, we did indeed not run on MR v1 for a while now. However, I don't
> > > want to get that far and say we don't support it anymore. I'll fix this
> > > particular issue by tomorrow.
> > >
> > > In the next couple of weeks we should run our full performance
> testsuite
> > > (for broad coverage) over an MR v1 cluster and systematically remove
> > > unnecessary incompatibility like this instance. Any volunteers?
> > >
> > > Regards,
> > > Matthias
> > >
> > > [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> > > PM---Hello, I got an error when running the
> > systemML/scripts/Univar-S]Ethan
> > > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running the
> > > systemML/scripts/Univar-Stats.dml script on
> > >
> > > From: Ethan Xu <et...@gmail.com>
> > > To: dev@systemml.incubator.apache.org
> > > Date: 02/04/2016 05:51 PM
> > > Subject: Compatibility with MR1 Cloudera cdh4.2.1
> > > ------------------------------
> >
> > >
> > >
> > >
> > > Hello,
> > >
> > > I got an error when running the systemML/scripts/Univar-Stats.dml
> script
> > on
> > > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message
> is
> > at
> > > the bottom of the email. The same script ran fine on a smaller sample
> > > (several MB) of the same data set, when MR was not invoked.
> > >
> > > The main error was java.lang.NoSuchMethodError:
> > > org.apache.hadoop.mapred.JobConf.getDouble()
> > > Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't
> > have
> > > the JobConf.getDouble() method.
> > >
> > > The hadoop-core jar of CDH4.2.1 can be found here:
> > >
> > >
> >
> >
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
> >
> > >
> > > The calling line of SystemML is line 1194 of
> > >
> > >
> >
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
> > >
> > > I was wondering, if the finding is accurate, is there a potential fix,
> or
> > > does this mean the current version of SystemML is not compatible with
> > > CDH4.2.1?
> > >
> > > Thank you,
> > >
> > > Ethan
> > >
> > >
> > > hadoop jar $sysDir/target/SystemML.jar -f
> > > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> > > X=$baseDirHDFS/original-coded.csv
> > > TYPES=$baseDirHDFS/original-coded-type.csv
> > > STATS=$baseDirHDFS/univariate-summary.csv
> > >
> > > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> > > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> > > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> > > (./SystemML-config.xml) found
> > > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
> > DMLConfig
> > > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> > > text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> > > (java.version=1.7.0_71).
> > > SLF4J: Class path contains multiple SLF4J bindings.
> > > SLF4J: Found binding in
> > >
> > >
> >
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: Found binding in
> > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: Found binding in
> > >
> > >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > > explanation.
> > > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> > > Total execution time: 0.880 sec.
> > > Number of executed MR Jobs: 0.
> > >
> > > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> > > Exception in thread "main" java.lang.NoSuchMethodError:
> > > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> > > at
> > >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> > > at
> > org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> > > at
> > >
> > >
> >
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> > > at
> > >
> > >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> > > at
> > >
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> > > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> > > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> > > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> > >
> > >
> > >
> >
> >
> >
>
--
Yifan "Ethan" Xu, PhD
Data Scientist / Statistician
Explorys, IBM Watson Health
Adjunct Faculty
Department of Epidemiology and Biostatistics
Case Western Reserve University
--------------
Email: ethan.yifanxu@gmail.com
Phone: (607) 760-6817
--------------
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Deron Eriksson <de...@gmail.com>.
Hi Matthias,
Glad to hear the fix is simple. Mixing jar versions sometimes is not very
fun.
Deron
On Thu, Feb 4, 2016 at 11:10 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
> well, let's not mix different hadoop versions in the class path or
> client/server. If I'm not mistaken, cdh 4.x always shipped with MR v1. It's
> a trivial fix for us and will be in the repo tomorrow morning anyway.
> Thanks for catching this issue Ethan.
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Deron Eriksson ---02/04/2016 11:04:38
> PM---Hi Ethan, Just FYI, I looked at hadoop-common-2.0.0-cdh4.2]Deron
> Eriksson ---02/04/2016 11:04:38 PM---Hi Ethan, Just FYI, I looked at
> hadoop-common-2.0.0-cdh4.2.1.jar (
>
> From: Deron Eriksson <de...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 02/04/2016 11:04 PM
> Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
> ------------------------------
>
>
>
> Hi Ethan,
>
> Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
> ),
> since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
> org.apache.hadoop.conf.Configuration class in that jar doesn't appear to
> have a getDouble method, so using that version of hadoop-common won't work.
>
> However, the hadoop-common-2.4.1.jar (
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
> )
>
> does appear to have the getDouble method. It's possible that adding that
> jar to your classpath may fix your problem, as Shirish pointed out.
>
> It sounds like Matthias may have another fix.
>
> Deron
>
>
>
> On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
>
> > well, we did indeed not run on MR v1 for a while now. However, I don't
> > want to get that far and say we don't support it anymore. I'll fix this
> > particular issue by tomorrow.
> >
> > In the next couple of weeks we should run our full performance testsuite
> > (for broad coverage) over an MR v1 cluster and systematically remove
> > unnecessary incompatibility like this instance. Any volunteers?
> >
> > Regards,
> > Matthias
> >
> > [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> > PM---Hello, I got an error when running the
> systemML/scripts/Univar-S]Ethan
> > Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running the
> > systemML/scripts/Univar-Stats.dml script on
> >
> > From: Ethan Xu <et...@gmail.com>
> > To: dev@systemml.incubator.apache.org
> > Date: 02/04/2016 05:51 PM
> > Subject: Compatibility with MR1 Cloudera cdh4.2.1
> > ------------------------------
>
> >
> >
> >
> > Hello,
> >
> > I got an error when running the systemML/scripts/Univar-Stats.dml script
> on
> > a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is
> at
> > the bottom of the email. The same script ran fine on a smaller sample
> > (several MB) of the same data set, when MR was not invoked.
> >
> > The main error was java.lang.NoSuchMethodError:
> > org.apache.hadoop.mapred.JobConf.getDouble()
> > Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't
> have
> > the JobConf.getDouble() method.
> >
> > The hadoop-core jar of CDH4.2.1 can be found here:
> >
> >
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>
> >
> > The calling line of SystemML is line 1194 of
> >
> >
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
> >
> > I was wondering, if the finding is accurate, is there a potential fix, or
> > does this mean the current version of SystemML is not compatible with
> > CDH4.2.1?
> >
> > Thank you,
> >
> > Ethan
> >
> >
> > hadoop jar $sysDir/target/SystemML.jar -f
> > $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> > X=$baseDirHDFS/original-coded.csv
> > TYPES=$baseDirHDFS/original-coded-type.csv
> > STATS=$baseDirHDFS/univariate-summary.csv
> >
> > 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> > 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> > 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> > (./SystemML-config.xml) found
> > 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
> DMLConfig
> > 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> > text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> > (java.version=1.7.0_71).
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> >
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> >
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> > Total execution time: 0.880 sec.
> > Number of executed MR Jobs: 0.
> >
> > 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> > Exception in thread "main" java.lang.NoSuchMethodError:
> > org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> > at
> >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> > at
> >
> >
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> > at
> >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> > at
> >
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> > at
> >
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> > at
> org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> > at
> >
> >
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> > at
> >
> >
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> > at
> >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> > at
> >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> > at
> >
> >
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> > at
> > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> > at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> >
> >
> >
>
>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Matthias Boehm <mb...@us.ibm.com>.
well, let's not mix different hadoop versions in the class path or
client/server. If I'm not mistaken, cdh 4.x always shipped with MR v1. It's
a trivial fix for us and will be in the repo tomorrow morning anyway.
Thanks for catching this issue Ethan.
Regards,
Matthias
From: Deron Eriksson <de...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 02/04/2016 11:04 PM
Subject: Re: Compatibility with MR1 Cloudera cdh4.2.1
Hi Ethan,
Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/
),
since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
org.apache.hadoop.conf.Configuration class in that jar doesn't appear to
have a getDouble method, so using that version of hadoop-common won't work.
However, the hadoop-common-2.4.1.jar (
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/
)
does appear to have the getDouble method. It's possible that adding that
jar to your classpath may fix your problem, as Shirish pointed out.
It sounds like Matthias may have another fix.
Deron
On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
> well, we did indeed not run on MR v1 for a while now. However, I don't
> want to get that far and say we don't support it anymore. I'll fix this
> particular issue by tomorrow.
>
> In the next couple of weeks we should run our full performance testsuite
> (for broad coverage) over an MR v1 cluster and systematically remove
> unnecessary incompatibility like this instance. Any volunteers?
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> PM---Hello, I got an error when running the
systemML/scripts/Univar-S]Ethan
> Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running the
> systemML/scripts/Univar-Stats.dml script on
>
> From: Ethan Xu <et...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 02/04/2016 05:51 PM
> Subject: Compatibility with MR1 Cloudera cdh4.2.1
> ------------------------------
>
>
>
> Hello,
>
> I got an error when running the systemML/scripts/Univar-Stats.dml script
on
> a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is
at
> the bottom of the email. The same script ran fine on a smaller sample
> (several MB) of the same data set, when MR was not invoked.
>
> The main error was java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble()
> Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't
have
> the JobConf.getDouble() method.
>
> The hadoop-core jar of CDH4.2.1 can be found here:
>
>
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>
> The calling line of SystemML is line 1194 of
>
>
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>
> I was wondering, if the finding is accurate, is there a potential fix, or
> does this mean the current version of SystemML is not compatible with
> CDH4.2.1?
>
> Thank you,
>
> Ethan
>
>
> hadoop jar $sysDir/target/SystemML.jar -f
> $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> X=$baseDirHDFS/original-coded.csv
> TYPES=$baseDirHDFS/original-coded-type.csv
> STATS=$baseDirHDFS/univariate-summary.csv
>
> 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> (./SystemML-config.xml) found
> 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in
DMLConfig
> 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.7.0_71).
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
>
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
>
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
>
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> Total execution time: 0.880 sec.
> Number of executed MR Jobs: 0.
>
> 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> at
>
>
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1195)
> at
>
>
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1129)
> at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:307)
> at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:289)
> at
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob
(CSVReblockMR.java:275)
> at org.apache.sysml.lops.runtime.RunMRJobs.submitJob
(RunMRJobs.java:257)
> at
>
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob
(RunMRJobs.java:143)
> at
>
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction
(MRJobInstruction.java:1500)
> at
>
>
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction
(ProgramBlock.java:309)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions
(ProgramBlock.java:227)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute
(ProgramBlock.java:169)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Deron Eriksson <de...@gmail.com>.
Hi Ethan,
Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/),
since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
org.apache.hadoop.conf.Configuration class in that jar doesn't appear to
have a getDouble method, so using that version of hadoop-common won't work.
However, the hadoop-common-2.4.1.jar (
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/)
does appear to have the getDouble method. It's possible that adding that
jar to your classpath may fix your problem, as Shirish pointed out.
It sounds like Matthias may have another fix.
Deron
On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mb...@us.ibm.com> wrote:
> well, we did indeed not run on MR v1 for a while now. However, I don't
> want to get that far and say we don't support it anymore. I'll fix this
> particular issue by tomorrow.
>
> In the next couple of weeks we should run our full performance testsuite
> (for broad coverage) over an MR v1 cluster and systematically remove
> unnecessary incompatibility like this instance. Any volunteers?
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> PM---Hello, I got an error when running the systemML/scripts/Univar-S]Ethan
> Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running the
> systemML/scripts/Univar-Stats.dml script on
>
> From: Ethan Xu <et...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 02/04/2016 05:51 PM
> Subject: Compatibility with MR1 Cloudera cdh4.2.1
> ------------------------------
>
>
>
> Hello,
>
> I got an error when running the systemML/scripts/Univar-Stats.dml script on
> a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
> the bottom of the email. The same script ran fine on a smaller sample
> (several MB) of the same data set, when MR was not invoked.
>
> The main error was java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble()
> Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
> the JobConf.getDouble() method.
>
> The hadoop-core jar of CDH4.2.1 can be found here:
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>
> The calling line of SystemML is line 1194 of
>
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>
> I was wondering, if the finding is accurate, is there a potential fix, or
> does this mean the current version of SystemML is not compatible with
> CDH4.2.1?
>
> Thank you,
>
> Ethan
>
>
> hadoop jar $sysDir/target/SystemML.jar -f
> $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> X=$baseDirHDFS/original-coded.csv
> TYPES=$baseDirHDFS/original-coded-type.csv
> STATS=$baseDirHDFS/univariate-summary.csv
>
> 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> (./SystemML-config.xml) found
> 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
> 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.7.0_71).
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> Total execution time: 0.880 sec.
> Number of executed MR Jobs: 0.
>
> 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
> at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
> at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
> at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
> at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
> at
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
> at org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
> at
>
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
> at
>
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
> at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
>
Re: Compatibility with MR1 Cloudera cdh4.2.1
Posted by Matthias Boehm <mb...@us.ibm.com>.
well, we did indeed not run on MR v1 for a while now. However, I don't want
to get that far and say we don't support it anymore. I'll fix this
particular issue by tomorrow.
In the next couple of weeks we should run our full performance testsuite
(for broad coverage) over an MR v1 cluster and systematically remove
unnecessary incompatibility like this instance. Any volunteers?
Regards,
Matthias
From: Ethan Xu <et...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 02/04/2016 05:51 PM
Subject: Compatibility with MR1 Cloudera cdh4.2.1
Hello,
I got an error when running the systemML/scripts/Univar-Stats.dml script on
a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
the bottom of the email. The same script ran fine on a smaller sample
(several MB) of the same data set, when MR was not invoked.
The main error was java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble()
Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
the JobConf.getDouble() method.
The hadoop-core jar of CDH4.2.1 can be found here:
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
The calling line of SystemML is line 1194 of
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
I was wondering, if the finding is accurate, is there a potential fix, or
does this mean the current version of SystemML is not compatible with
CDH4.2.1?
Thank you,
Ethan
hadoop jar $sysDir/target/SystemML.jar -f
$sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
X=$baseDirHDFS/original-coded.csv
TYPES=$baseDirHDFS/original-coded-type.csv
STATS=$baseDirHDFS/univariate-summary.csv
16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
(./SystemML-config.xml) found
16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
text read for 'text' and 'csv' due to thread contention on JRE < 1.8
(java.version=1.7.0_71).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
Total execution time: 0.880 sec.
Number of executed MR Jobs: 0.
16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1195)
at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1129)
at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:307)
at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:289)
at
org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
at org.apache.sysml.lops.runtime.RunMRJobs.submitJob
(RunMRJobs.java:257)
at
org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob
(RunMRJobs.java:143)
at
org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction
(MRJobInstruction.java:1500)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction
(ProgramBlock.java:309)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions
(ProgramBlock.java:227)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute
(ProgramBlock.java:169)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)