You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Danny Bickson (JIRA)" <ji...@apache.org> on 2011/01/26 04:02:44 UTC
[jira] Issue Comment Edited: (MAHOUT-542) MapReduce implementation
of ALS-WR
[ https://issues.apache.org/jira/browse/MAHOUT-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986825#action_12986825 ]
Danny Bickson edited comment on MAHOUT-542 at 1/25/11 10:01 PM:
----------------------------------------------------------------
Hi!
I tried to install patch MAHOUT-542-3 against a clean svn checkout from Mahout trunk.
it seems that the file AlternateLeastSquaresSolver.java was already inserted into svn, so the code got doubled.
I tried to edit manually changes, and I attach a patch I created (named unifiedpatch)
The code compiles, but fails some tests.
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.mahout.math.jet.random.NormalTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.437 sec
Running org.apache.mahout.math.TestMatrixView
Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.283 sec
Running org.apache.mahout.math.TestDenseVector
Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.134 sec
Running org.apache.mahout.math.TestSparseMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.315 sec
Running org.apache.mahout.math.jet.stat.ProbabilityTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.162 sec
Running org.apache.mahout.math.VectorListTest
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.291 sec
Running org.apache.mahout.math.stats.LogLikelihoodTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec
Running org.apache.mahout.math.jet.random.engine.MersenneTwisterTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 sec
Running org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.071 sec
Running org.apache.mahout.math.TestDenseMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec
Running org.apache.mahout.math.jet.random.ExponentialTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.707 sec
Running org.apache.mahout.math.jet.stat.GammaTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.193 sec
Running org.apache.mahout.math.stats.OnlineSummarizerTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec
Running org.apache.mahout.math.VectorTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec
Running org.apache.mahout.common.RandomUtilsTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec
Running org.apache.mahout.math.TestSparseColumnMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.104 sec
Running org.apache.mahout.math.TestSequentialAccessSparseVector
Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec
Running org.apache.mahout.math.QRDecompositionTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.045 sec
Running org.apache.mahout.math.TestSparseRowMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.09 sec
Running org.apache.mahout.math.TestRandomAccessSparseVector
Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec
Running org.apache.mahout.math.TestOrderedIntDoubleMapping
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.apache.mahout.math.als.AlternateLeastSquaresSolverTest
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec <<< FAILURE!
Running org.apache.mahout.math.jet.random.NegativeBinomialTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec
Running org.apache.mahout.math.TestVectorView
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec
Running org.apache.mahout.math.jet.random.GammaTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.72 sec
Running org.apache.mahout.math.TestSingularValueDecomposition
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec
Running org.apache.mahout.math.decomposer.lanczos.TestLanczosSolver
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.875 sec
Results :
Failed tests:
createRiIiMaybeTransposedExceptionOnNonSequentialVector(org.apache.mahout.math.als.AlternateLeastSquaresSolverTest)
ubuntu@ip-10-195-226-63:/usr/local/mahout-0.4$ cat math/target/surefire-reports/org.apache.mahout.math.als.AlternateLeastSquaresSolverTest.txt
-------------------------------------------------------------------------------
Test set: org.apache.mahout.math.als.AlternateLeastSquaresSolverTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec <<< FAILURE!
createRiIiMaybeTransposedExceptionOnNonSequentialVector(org.apache.mahout.math.als.AlternateLeastSquaresSolverTest) Time elapsed: 0.002 sec <<< FAILURE!
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.fail(Assert.java:98)
at org.apache.mahout.math.als.AlternateLeastSquaresSolverTest.createRiIiMaybeTransposedExceptionOnNonSequentialVector(AlternateLeastSquaresSolverTest.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Can you please take a look and instruct me what to do?
Thanks,
Danny Bickson
was (Author: danny.bickson):
Hi!
I tried to install patch MAHOUT-542 against a clean svn checkout from Mahout trunk.
it seems that the file AlternateLeastSquaresSolver.java was already inserted into svn, so the code got doubled.
I tried to edit manually changes, and I attach a patch I created (named unifiedpatch)
The code compiles, but fails some tests.
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.mahout.math.jet.random.NormalTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.437 sec
Running org.apache.mahout.math.TestMatrixView
Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.283 sec
Running org.apache.mahout.math.TestDenseVector
Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.134 sec
Running org.apache.mahout.math.TestSparseMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.315 sec
Running org.apache.mahout.math.jet.stat.ProbabilityTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.162 sec
Running org.apache.mahout.math.VectorListTest
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.291 sec
Running org.apache.mahout.math.stats.LogLikelihoodTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec
Running org.apache.mahout.math.jet.random.engine.MersenneTwisterTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 sec
Running org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.071 sec
Running org.apache.mahout.math.TestDenseMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec
Running org.apache.mahout.math.jet.random.ExponentialTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.707 sec
Running org.apache.mahout.math.jet.stat.GammaTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.193 sec
Running org.apache.mahout.math.stats.OnlineSummarizerTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec
Running org.apache.mahout.math.VectorTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec
Running org.apache.mahout.common.RandomUtilsTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec
Running org.apache.mahout.math.TestSparseColumnMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.104 sec
Running org.apache.mahout.math.TestSequentialAccessSparseVector
Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec
Running org.apache.mahout.math.QRDecompositionTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.045 sec
Running org.apache.mahout.math.TestSparseRowMatrix
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.09 sec
Running org.apache.mahout.math.TestRandomAccessSparseVector
Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec
Running org.apache.mahout.math.TestOrderedIntDoubleMapping
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.apache.mahout.math.als.AlternateLeastSquaresSolverTest
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec <<< FAILURE!
Running org.apache.mahout.math.jet.random.NegativeBinomialTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec
Running org.apache.mahout.math.TestVectorView
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec
Running org.apache.mahout.math.jet.random.GammaTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.72 sec
Running org.apache.mahout.math.TestSingularValueDecomposition
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec
Running org.apache.mahout.math.decomposer.lanczos.TestLanczosSolver
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.875 sec
Results :
Failed tests:
createRiIiMaybeTransposedExceptionOnNonSequentialVector(org.apache.mahout.math.als.AlternateLeastSquaresSolverTest)
ubuntu@ip-10-195-226-63:/usr/local/mahout-0.4$ cat math/target/surefire-reports/org.apache.mahout.math.als.AlternateLeastSquaresSolverTest.txt
-------------------------------------------------------------------------------
Test set: org.apache.mahout.math.als.AlternateLeastSquaresSolverTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec <<< FAILURE!
createRiIiMaybeTransposedExceptionOnNonSequentialVector(org.apache.mahout.math.als.AlternateLeastSquaresSolverTest) Time elapsed: 0.002 sec <<< FAILURE!
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.fail(Assert.java:98)
at org.apache.mahout.math.als.AlternateLeastSquaresSolverTest.createRiIiMaybeTransposedExceptionOnNonSequentialVector(AlternateLeastSquaresSolverTest.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Can you please take a look and instruct me what to do?
Thanks,
Danny Bickson
> MapReduce implementation of ALS-WR
> ----------------------------------
>
> Key: MAHOUT-542
> URL: https://issues.apache.org/jira/browse/MAHOUT-542
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Affects Versions: 0.5
> Reporter: Sebastian Schelter
> Attachments: MAHOUT-452.patch, MAHOUT-542-2.patch, MAHOUT-542-3.patch, unifiedpatch
>
>
> As Mahout is currently lacking a distributed collaborative filtering algorithm that uses matrix factorization, I spent some time reading through a couple of the Netflix papers and stumbled upon the "Large-scale Parallel Collaborative Filtering for the Netflix Prize" available at http://www.hpl.hp.com/personal/Robert_Schreiber/papers/2008%20AAIM%20Netflix/netflix_aaim08(submitted).pdf.
> It describes a parallel algorithm that uses "Alternating-Least-Squares with Weighted-λ-Regularization" to factorize the preference-matrix and gives some insights on how the authors distributed the computation using Matlab.
> It seemed to me that this approach could also easily be parallelized using Map/Reduce, so I sat down and created a prototype version. I'm not really sure I got the mathematical details correct (they need some optimization anyway), but I wanna put up my prototype implementation here per Yonik's law of patches.
> Maybe someone has the time and motivation to work a little on this with me. It would be great if someone could validate the approach taken (I'm willing to help as the code might not be intuitive to read) and could try to factorize some test data and give feedback then.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.