You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Kai Hui (JIRA)" <ji...@apache.org> on 2015/09/08 17:22:46 UTC
[jira] [Comment Edited] (MAHOUT-1408) Distributed cache file
matching bug while running SSVD in broadcast mode
[ https://issues.apache.org/jira/browse/MAHOUT-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14734981#comment-14734981 ]
Kai Hui edited comment on MAHOUT-1408 at 9/8/15 3:22 PM:
---------------------------------------------------------
I am also experiencing the same problem at this moment.. What I did is simiply call the toolrunner.run(SSVDCli, args) within my class.. The reason to do that is due to I cant install the mahout on the server due to the edition of the maven and I have no sudo ...
And I received exactly the same error, which I guess is due to there exist tmp files in distributed cash for follow-up pipeline in ssvd solver... However, there exist other files, like jars in the distributed cash, and all these other files cant pass thru the pattern checking, i.e., not following the specific patter requried in SSVDHelper$1.compare(SSVDHelper.java:152). And the whole job failed... I would suppose a file name filter before the pattern matcher would do the job...
For the future reference, I wrote down my solution as follows and it worked for me: add the jar files to the class path inside your java code with addArchiveToClassPath instead of using -libjars in the command line of hadoop jar ...:
FileSystem fs = FileSystem.get(conf);
String libdir = "the hdfs uri for your jar, like /user/<username>/libjars"
FileStatus[] status_list = fs.listStatus(new Path(libdir));
if (status_list != null) {
for (FileStatus status : status_list) {
String fname = status.getPath().getName();
if (fname.endsWith("jar")) {
Path path2add = new Path(libdir + "/" + fname);
DistributedCache.addArchiveToClassPath(path2add, conf, fs);
}
}
}
was (Author: lhyan792):
I am also experiencing the same problem at this moment.. What I did is simiply call the toolrunner.run(SSVDCli, args) within my class.. The reason to do that is due to I cant install the mahout on the server due to the edition of the maven and I have no sudo ...
And I received exactly the same error, which I guess is due to there exist tmp files in distributed cash for follow-up pipeline in ssvd solver... However, there exist other files, like jars in the distributed cash, and all these other files cant pass thru the pattern checking, i.e., not following the specific patter of the file name {filename}-{p}-{number} requried in SSVDHelper$1.compare(SSVDHelper.java:152). And the whole job failed... I would suppose a file name filter before the pattern matcher would do the job...
For the future reference, I wrote down my solution as follows and it worked for me: add the jar files to the class path inside your java code with addArchiveToClassPath instead of using -libjars in the command line of hadoop jar ...:
FileSystem fs = FileSystem.get(conf);
String libdir = "the hdfs uri for your jar, like /user/<username>/libjars"
FileStatus[] status_list = fs.listStatus(new Path(libdir));
if (status_list != null) {
for (FileStatus status : status_list) {
String fname = status.getPath().getName();
if (fname.endsWith("jar")) {
Path path2add = new Path(libdir + "/" + fname);
DistributedCache.addArchiveToClassPath(path2add, conf, fs);
}
}
}
> Distributed cache file matching bug while running SSVD in broadcast mode
> ------------------------------------------------------------------------
>
> Key: MAHOUT-1408
> URL: https://issues.apache.org/jira/browse/MAHOUT-1408
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.8
> Reporter: Angad Singh
> Assignee: Dmitriy Lyubimov
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments: BtJob.java.patch
>
>
> The error is:
> java.lang.IllegalArgumentException: Unexpected file name, unable to deduce partition #:file:/data/d1/mapred/local/taskTracker/distcache/434503979705629827_-1822139941_1047712745/nn.red.ua2.inmobi.com/user/rmcuser/oozie-oozi/0034272-140120102756143-oozie-oozi-W/inmobi-ssvd_mahout--java/java-launcher.jar
> at org.apache.mahout.math.hadoop.stochasticsvd.SSVDHelper$1.compare(SSVDHelper.java:154)
> at org.apache.mahout.math.hadoop.stochasticsvd.SSVDHelper$1.compare(SSVDHelper.java:1)
> at java.util.Arrays.mergeSort(Arrays.java:1270)
> at java.util.Arrays.mergeSort(Arrays.java:1281)
> at java.util.Arrays.mergeSort(Arrays.java:1281)
> at java.util.Arrays.sort(Arrays.java:1210)
> at org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator.init(SequenceFileDirValueIterator.java:112)
> at org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator.<init>(SequenceFileDirValueIterator.java:94)
> at org.apache.mahout.math.hadoop.stochasticsvd.BtJob$BtMapper.setup(BtJob.java:220)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> The bug is @ https://github.com/apache/mahout/blob/trunk/core/src/main/java/org/apache/mahout/math/hadoop/stochasticsvd/BtJob.java, near line 220.
> and @ https://github.com/apache/mahout/blob/trunk/core/src/main/java/org/apache/mahout/math/hadoop/stochasticsvd/SSVDHelper.java near line 144.
> SSVDHelper's PARTITION_COMPARATOR assumes all files in the distributed cache will have a particular pattern whereas we have jar files in our distributed cache which causes the above exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)