You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Drew Farris (JIRA)" <ji...@apache.org> on 2010/03/03 05:53:27 UTC
[jira] Created: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
--------------------------------------------------------------------------------------------
Key: MAHOUT-320
URL: https://issues.apache.org/jira/browse/MAHOUT-320
Project: Mahout
Issue Type: Improvement
Affects Versions: 0.3
Reporter: Drew Farris
Priority: Minor
Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Drew Farris updated MAHOUT-320:
-------------------------------
Assignee: Robin Anil
Status: Patch Available (was: Open)
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841181#action_12841181 ]
Sean Owen commented on MAHOUT-320:
----------------------------------
After the fact here, but I'm going to fix a number of small issues in IntPairWritable, and three that I think are important enough to call out. I keep seeing these kinds of subtle bugs:
- hashCode() and equals() not only must both be implemented, but must be consistent. Frequency uses 'pair' in hashCode() but not in equals()
- and you can hash doubles much faster with RandomUtils.hashDouble()
- compareTo() must be consistent with equals(). It returns 0 exactly when equals() returns true. Frequency.compareTo() can never return 0, which is never correct, even if only because this breaks symmetry
- compareTo() that decides ordering based on two integer values should never be implemented with a subtraction, unless you're absolutely certain the following can't occur: trying to order a=Integer.MAX_VALUE and b=Integer.MIN_VALUE will put a before b, if you base it on a-b's value, since it overflows to negative.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-320:
-----------------------------
Attachment: IntPairWritable.patch
I see. It's to make positive numbers negative and vice versa, in order to use WritableComparable's compare() function on bytes, which assumes values are essentially unsigned. Surely this will end in tears to store values this way. In particular it's already broken Frequency in the same class, which reads the values as unsigned ints directly.
Here's my complete patch for all said items.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: IntPairWritable.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840560#action_12840560 ]
Sean Owen commented on MAHOUT-320:
----------------------------------
Oh what are we referring to by 'binary'?
Bigram has some bits like this that seem to be able to compare based on a byte representation -- assuming that's a nice optimization within Hadoop:
@Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
int ret;
try {
int firstb1 = WritableComparator.readVInt(b1, s1);
int firstb2 = WritableComparator.readVInt(b2, s2);
ret = firstb1 - firstb2;
} catch (IOException ioe) {
throw new IllegalArgumentException(ioe);
}
return ret;
}
(Though we gotta fix this returning firstb1 - firstb2 thing -- overflow makes this result incorrect for about 1 in 16 possible pair values!)
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840530#action_12840530 ]
Sean Owen commented on MAHOUT-320:
----------------------------------
On a related note, as Robin mentioned, this class could be merged with Bigram, or vice versa. I don't think ankur would be too bothered (?) about replacing Bigram, but, it actually seems to be more well developed. It does binary comparison too for instance. Might this patch take the extra step of merging the two?
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robin Anil updated MAHOUT-320:
------------------------------
Attachment: MAHOUT-320.patch
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840601#action_12840601 ]
Ankur commented on MAHOUT-320:
------------------------------
Binary comparison looks more or less the same in both the classes. Its the data serialization/serialization where Bigram scores over IntPairWritable. Bigram encodes/decodes the data in VInt format which uses zero compressed encodings for more info see o.a.h.io.WritableUtils.java. The encoding can give considerable savings when serializing huge amounts of numeric data.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Drew Farris updated MAHOUT-320:
-------------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.4
Resolution: Fixed
Looks like this one has been resolved for awhile.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Fix For: 0.4
>
> Attachments: IntPairWritable.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robin Anil updated MAHOUT-320:
------------------------------
Attachment: MAHOUT-320.patch
Has my tweaks in LDA too along with removal of bigram
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841187#action_12841187 ]
Sean Owen commented on MAHOUT-320:
----------------------------------
While I'm here, what's the point in the "^ 0x80" express in get/putInt()? It flips the top bit when writing, and unflips when reading, which seems like a confusing no-op. What am I missing?
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840553#action_12840553 ]
Robin Anil commented on MAHOUT-320:
-----------------------------------
This too does binary comparison right. compare by first then second
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robin Anil updated MAHOUT-320:
------------------------------
Attachment: MAHOUT-320.patch
Oops missed out the changes in LDA Tests. Here you go
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840634#action_12840634 ]
Ankur commented on MAHOUT-320:
------------------------------
I can't apply the patch cleanly. Here is what it says
can't find file to patch at input line 1285
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|Index: core/src/main/java/org/apache/mahout/common/IntPairWritable.java
|===================================================================
|--- core/src/main/java/org/apache/mahout/common/IntPairWritable.java (revision 918394)
|+++ core/src/main/java/org/apache/mahout/common/IntPairWritable.java (working copy)
--------------------------
File to patch:
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Drew Farris updated MAHOUT-320:
-------------------------------
Component/s: Clustering
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840593#action_12840593 ]
Sean Owen commented on MAHOUT-320:
----------------------------------
Good point. That's what Bigram is doing. Ankur any thoughts?
I looove consolidation of code.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Drew Farris updated MAHOUT-320:
-------------------------------
Attachment: MAHOUT-320.patch
binary comparable implementation plus unit test for get/set, writable methods and comparator.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841059#action_12841059 ]
Ankur commented on MAHOUT-320:
------------------------------
It still complains that it cannot find the file to patch - core/src/main/java/org/apache/mahout/common/IntPairWritable.java. Also it looks like the unti test for IntPairWritable is still lying under o.a.m.clustering.lda.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robin Anil updated MAHOUT-320:
------------------------------
Attachment: MAHOUT-320.patch
new patch from trunk
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840564#action_12840564 ]
Robin Anil commented on MAHOUT-320:
-----------------------------------
This was what intpairwritable does now. This patch has a faster comparison. Byte based
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840620#action_12840620 ]
Robin Anil commented on MAHOUT-320:
-----------------------------------
Yeah VInt Does give savings. but the overhead i see in LDA shows that the same saving if not more can be achieved by output compression. I have just integrated the two. and moved it to common. Since there is no test for co-occurrence, I cannot verify if i dint break it or not.
Ankut can you just verify as soon as i upload the patch
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840575#action_12840575 ]
Sean Owen commented on MAHOUT-320:
----------------------------------
I can see that at the end now. How do you feel about replacing Bigram with this and moving it to a common package? Also how about using Bigram's variable-length encoding? It might be more efficient as well as avoid you writing routines to serialize an int to a byte stream.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840592#action_12840592 ]
Drew Farris commented on MAHOUT-320:
------------------------------------
The big win here is sortability of the binary form. I'm not certain that IntPairWritables that used variable length encoding would be inherently sortable without converting them back to ints.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840695#action_12840695 ]
Robin Anil commented on MAHOUT-320:
-----------------------------------
Any issues or does it look good to commit ?
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840636#action_12840636 ]
Ankur commented on MAHOUT-320:
------------------------------
Robin, Can you update your revision and create a fresh patch ?
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Ankur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840619#action_12840619 ]
Ankur commented on MAHOUT-320:
------------------------------
And yes I see the issue with (firstb1 - firstb2) thing in Bigram. This definitely needs to be fixed. I don't mind replacing one with either. Just that we should be using VInt format for ser/de on the wire.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Robin Anil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841065#action_12841065 ]
Robin Anil commented on MAHOUT-320:
-----------------------------------
I will commit the LDA part with IntWritable, and leave out bigram. Create another patch just for co-occurrence
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841220#action_12841220 ]
Drew Farris commented on MAHOUT-320:
------------------------------------
I haven't reviewed the round of patches, when I wrote the initial patch the '^ 0x80' was in there so that the byte array holding the ints would be sorted in natural int order.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch, MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-320) Modify IntPairWritable in LDA
implementation to be binary comparable to improve performance.
Posted by "Drew Farris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840617#action_12840617 ]
Drew Farris commented on MAHOUT-320:
------------------------------------
I certainlly can't argure about the space savings. VInts are definitely more efficient in that sense. However, the conversion to/from VInts gets expensive in the sense that during the sort, Bigram is performing multiple decodes per pair that is compared. The question that remains is whether the cost of slinging the extra data around mitigates the gain from not doing those decodes in IntPairWritable.
> Modify IntPairWritable in LDA implementation to be binary comparable to improve performance.
> --------------------------------------------------------------------------------------------
>
> Key: MAHOUT-320
> URL: https://issues.apache.org/jira/browse/MAHOUT-320
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.3
> Reporter: Drew Farris
> Assignee: Robin Anil
> Priority: Minor
> Attachments: MAHOUT-320.patch
>
>
> Per discussion with Robin, modifying o.a.m.clustering.lda.IntPairWritable to be binary comparable will improve the performance of the comparison operations during a sort because no marshaling will need to occur to compare IntPairWritable instances.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.