[jira] [Work logged] (MAHOUT-468) Performance of RowSimilarityJob is not good

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAHOUT-468?focusedWorklogId=941980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-941980 ]

ASF GitHub Bot logged work on MAHOUT-468:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Nov/24 21:32
            Start Date: 05/Nov/24 21:32
    Worklog Time Spent: 10m 
      Work Description: rawkintrevo opened a new pull request, #472:
URL: https://github.com/apache/mahout/pull/472

   ### Purpose of PR:
   Please give a short description of what this PR is for.
   #468 Add a parameter object
   #469 Bind Parameters at execution
   with bonus notebook for optimization loop external to qumat
   
   
   ### Important ToDos
   Please mark each with an "x"
   - [x] A JIRA ticket exists (if not, please create this first)[https://issues.apache.org/jira/browse/mahout/]
   - [x] Title of PR is "MAHOUT-XXXX Brief Description of Changes" where XXXX is the JIRA number.
   - [ ] Created unit tests where appropriate
   
   - [x] Assigned JIRA to self
   - [x] Added documentation in scala docs/java docs, and to website
   
   https://colab.research.google.com/drive/1QgeqKZtkC_hOmffRprxJWMcC1g9YsMOE?usp=sharing qiskit
   https://colab.research.google.com/drive/1M5tT7h2wRiGNYx48g_FDGPTKuogKxEzc?usp=sharing cirq
   
   bracket untested
   
   Does this change break earlier versions?
   no
   
   Is this the beginning of a larger project for which a feature branch should be made?
   no
   




Issue Time Tracking
-------------------

            Worklog Id:     (was: 941980)
    Remaining Estimate: 0h
            Time Spent: 10m

> Performance of RowSimilarityJob is not good
> -------------------------------------------
>
>                 Key: MAHOUT-468
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-468
>             Project: Mahout
>          Issue Type: Test
>    Affects Versions: 0.4
>            Reporter: Han Hui Wen 
>            Priority: Major
>         Attachments: RowSimilarityJob-CooccurrencesMapper-SimilarityReducer.jpg
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> I have done a test ,
> Preferences records: 680,194
> distinct users: 23,246
> distinct items:437,569 
> SIMILARITY_CLASS_NAME=SIMILARITY_COOCCURRENCE
> maybePruneItemUserMatrixPath:16.50M
> weights:13.80M
> pairwiseSimilarity:18.81G
> Job RowSimilarityJob-RowWeightMapper-WeightedOccurrencesPerColumnReducer:used 32 sec
> Job RowSimilarityJob-CooccurrencesMapper-SimilarityReducer:used 4.30 hours
> I think the reason may be following:
> 1) We used SequenceFileOutputFormat,it cause job can only be run by n ( n= Hadoop node counts ) mappers or reducers concurrently.
> 2)  We stored redundant info.
> for example :
> the output of CooccurrencesMapper: (ItemIndexA,similarity),(ItemIndexA,ItemIndexB,similarity)
> 3) Some frequently used code 
> https://issues.apache.org/jira/browse/MAHOUT-467
> 4) allocate many local variable in loop (need confirm )
> In Class DistributedUncenteredZeroAssumingCosineVectorSimilarity
>   @Override
>   public double weight(Vector v) {
>     double length = 0.0;
>     Iterator<Element> elemIterator = v.iterateNonZero();
>     while (elemIterator.hasNext()) {
>       double value = elemIterator.next().get();  //this one
>       length += value * value;
>     }
>     return Math.sqrt(length);
>   }
> 5) Maybe we need control the size of cooccurrences



--
This message was sent by Atlassian Jira
(v8.20.10#820010)