Using Mahout 1.0-SNAPSHOT with yarn cluster

[mw <mw...@plista.com>]

Hello,

i am working on a web application that should execute lda on a external 
yarn cluster.

I am uploading all the relevant sequence files onto the yarn cluter.
This is how it try to remotely execute lda on the cluster.

         try {
             ugi.doAs(new PrivilegedExceptionAction<Void>() {
                 public Void run() throws Exception {
                     Configuration hdoopConf = new Configuration();
                     hdoopConf.set("fs.defaultFS", 
"hdfs://xxx.xxx.xxx.xxx:9000/user/xx");
                     hdoopConf.set("yarn.resourcemanager.hostname", 
"xxx.xxx.xxx.xxx");
                     hdoopConf.set("mapreduce.framework.name", "yarn");
                     hdoopConf.set("mapred.framework.name", "yarn");
                     hdoopConf.set("mapred.job.tracker", "xxx.xxx.xxx.xxx");
                     hdoopConf.set("dfs.permissions.enabled", "false");
                     hdoopConf.set("hadoop.job.ugi", "xx");
hdoopConf.set("mapreduce.jobhistory.address","xxx.xxx.xxx.xxx:10020" );
                     CVB0Driver driver = new CVB0Driver();
                     try {
                         driver.run(hdoopConf, 
sparseVectorIn.suffix("/matrix"),
                                 topicsOut, k, numTerms, 
doc_topic_smoothening, term_topic_smoothening,
                                 maxIter, iteration_block_size, 
convergenceDelta,
sparseVectorIn.suffix("/dictionary.file-0"), 
topicsOut.suffix("/DocumentTopics/"), sparseVectorIn,
                                 seed, testFraction, numTrainThreads, 
numUpdateThreads, maxItersPerDoc,
                                 numReduceTasks, backfillPerplexity);
                     } catch (ClassNotFoundException e) {
                         e.printStackTrace();
                     } catch (InterruptedException e) {
                         e.printStackTrace();
                     }
                     return null;
                 }
             });
         } catch (InterruptedException e) {
             e.printStackTrace();
         }

I am getting the following error message:

java.lang.InterruptedException: Failed to complete iteration 1 stage 1
     at 
org.apache.mahout.clustering.lda.cvb.CVB0Driver.runIteration(CVB0Driver.java:502)
     at 
org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:319)
     at 
org.plista.kornakapi.core.training.LDATopicFactorizer$1.run(LDATopicFactorizer.java:227)
     at 
org.plista.kornakapi.core.training.LDATopicFactorizer$1.run(LDATopicFactorizer.java:208)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:422)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
     ...

Is my general setup correct or did i understood something wrong?

Best,

Max