You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2009/08/11 14:59:15 UTC

[jira] Created: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Added support for mapping String to long IDs in CF code
-------------------------------------------------------

                 Key: MAHOUT-162
                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
             Project: Mahout
          Issue Type: Improvement
          Components: Collaborative Filtering
    Affects Versions: 0.2
            Reporter: Sean Owen
            Assignee: Sean Owen
             Fix For: 0.2


Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:

- Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
- Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Re: [jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by Ted Dunning <te...@gmail.com>.
I feel for you (a little bit).  I went through the same change last year.

In fact, it was painless.

If you plan to use the hadoop based components of mahout, you should look
into using 1.6 for more reasons than just this one getBytes call.

On Sun, Aug 23, 2009 at 10:08 PM, Weiwei Shi (JIRA) <ji...@apache.org> wrote:

> It won't be easy for us to migrate the production env to java 6. We will
> have to use our own IDMirgrator implementation then.




-- 
Ted Dunning, CTO
DeepDyve

[jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743256#action_12743256 ] 

Otis Gospodnetic commented on MAHOUT-162:
-----------------------------------------

Had a quick look and this seems nice and straight forward to me.  Commit? :)


> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved MAHOUT-162.
------------------------------

    Resolution: Fixed

Committed patch as it seems to work for me (and Otis I presume) and heard no objections.

> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Weiwei Shi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746731#action_12746731 ] 

Weiwei Shi commented on MAHOUT-162:
-----------------------------------

It won't be easy for us to migrate the production env to java 6. We will have to use our own IDMirgrator implementation then.

> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Weiwei Shi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746717#action_12746717 ] 

Weiwei Shi commented on MAHOUT-162:
-----------------------------------

In AbstractIDMigrator.java I see this.

private static final Charset UTF8_CHARSET = Charset.forName("UTF8");
...
md5hash = md5Digest.digest(value.getBytes(UTF8_CHARSET));

It would be nice if it uses the Java 5 compatible method String.getBytes(String) instead of String.getBytes(Charset)

> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746724#action_12746724 ] 

Ted Dunning commented on MAHOUT-162:
------------------------------------


Mahout uses java 1.6 where there is an advantage to do so.  This requirement is inherited from Hadoop.

This code that you mention is a good example of an additional small improvement available with 1.6.  The java 5 version of getBytes throws an UnsupportedEncodingException which is a checked exception.  Since there isn't much that you can do about this, that is just pain for no gain.



> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746770#action_12746770 ] 

Sean Owen commented on MAHOUT-162:
----------------------------------

Doesn't a lot more than this fail in Java 6, like @Override on implemented interface methods? or are you compiling with 6 and running on 5? If so, is this really the only incompatible difference at runtime? Hmm, maybe so.

I do echo Ted's comment that switching to Java 6 is a good idea but understand that is sometimes easier said than done. Since the project is officially on Java 6, I'd hesitate to have an informal requirement to not use any new APIs in 6. I do agree, this was a good API change that avoids an unnecessary and annoying exception. Still, one could say, it's such a small change to create backwards compatibility.

I'm going to suggest you simply reuse and change the code for Java 5 for now, and if you move to Java 6 later, you can quite easily use the standard implementation.

> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated MAHOUT-162:
-----------------------------

    Attachment: MAHOUT-162.patch

Draft patch.

> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAHOUT-162) Added support for mapping String to long IDs in CF code

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743375#action_12743375 ] 

Sean Owen commented on MAHOUT-162:
----------------------------------

The interesting part might be the choice of hash function. I reversed myself and went with MD5. The reason is that it is more portable since it is a well known function implemented in many contexts and this could aid integration. For example from a MySQL database table using string ids, once can simply select 'conv(substr(md5(id),1,16),16,10)' instead of 'id' from the table and that is about it. 

Good glad it looks pretty straightforward. My laptop died so it will be some time before I can rebuild and commit from a new machine. Good thing I put out the patch, that is about the only thing I hadn't saved. 

> Added support for mapping String to long IDs in CF code
> -------------------------------------------------------
>
>                 Key: MAHOUT-162
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-162
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.2
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>             Fix For: 0.2
>
>         Attachments: MAHOUT-162.patch
>
>
> Since the framework now only allows long (64-bit integer) IDs, and no longer Strings, we need to provide some support for translating between the two. The basic proposal is this:
> - Define a one-way mapping from Strings to longs that is repeatable and easy to implement in many contexts. In particular I propose using the bottom 64 bits of the MD5 hash of a string.
> - Define support for storing the reverse mapping (longs to Strings) in various ways, in an efficient way, that handles gracefully the very rare possibility of collision

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.