[jira] Commented: (MAHOUT-158) Replace all ID values with long

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740070#action_12740070 ] 

Sean Owen commented on MAHOUT-158:
----------------------------------

For the interested, was able to drive the memory requirements down to more like the expected value -- now running comfortably in a heap of 360M compared to needing 600M before (and more like 1GB before MAHOUT-151/154).

It was an interesting lesson in GC ergonomics. I found myself running into incredible GC overhead before the heap was full -- not even close. I learned the difference between the young generation and tenured generation in the GC: the default way memory is organized, it will let "old" objects consume only about 75% of the heap. Now that this system is more lean, almost all objects in memory are long-lived, and a lot less garbage is generated since long primitives are used instead of Longs and there is much less conversion between the two. So I had to set -XX:NewRatio=9 to ask it to allow more like 90% for 'old' objects. Then I was able to bring down the heap size to a more reasonble value.

I am proceeding to convert the tests now, as I review the changes. This is another big one.

> Replace all ID values with long
> -------------------------------
>
>                 Key: MAHOUT-158
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-158
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Clustering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-158.patch
>
>
> As mentioned on mailing list, I am tracking this as a possible change for evaluation. The idea is to save more memory / CPU by avoiding the Object overhead of tens of millions of ID objects by using long IDs instead.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.