You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2012/10/14 20:53:02 UTC
[jira] [Created] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Josh Wills created CRUNCH-96:
--------------------------------
Summary: Add secondary sort functionality to o.a.c.lib
Key: CRUNCH-96
URL: https://issues.apache.org/jira/browse/CRUNCH-96
Project: Crunch
Issue Type: Improvement
Components: Core, MapReduce Patterns
Reporter: Josh Wills
Assignee: Josh Wills
Fix For: 0.4.0
Attachments: CRUNCH-96.patch
I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477994#comment-13477994 ]
Josh Wills commented on CRUNCH-96:
----------------------------------
Yeah, I'm +1 for lib.sort, esp. as we're doing lib.text in CRUNCH-97. Will throw together another patch this evening.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478274#comment-13478274 ]
Josh Wills commented on CRUNCH-96:
----------------------------------
I ran into it on a machine learning project I was working on, and it seems to come up fairly often in sessionization applications (e.g., group by user ID, sort events by timestamp), viz., https://www.google.com/search?q=mapreduce+secondary+sort
Your point on naming well-taken: this isn't a total ordering on the keys, it's just a sort on the values going into the reducer. Something more like GroupByKeyWithSecondarySort would be more accurate (albeit more verbose.) Recommendations?
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478298#comment-13478298 ]
Gabriel Reid commented on CRUNCH-96:
------------------------------------
Ok, got it. I didn't realize that SecondarySort was a well-known name, so we should probably just leave it as-is, and then just change the SecondarySort in the examples module to SecondarySortExample.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480916#comment-13480916 ]
Gabriel Reid commented on CRUNCH-96:
------------------------------------
Looks good to me, I'd say commit it.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96b.patch, CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478223#comment-13478223 ]
Gabriel Reid commented on CRUNCH-96:
------------------------------------
Just took a look through this. Out of curiosity, is there that much of a use case specifically for this? I'm definitely not against adding it, but I'm also wondering if this is a really specific use case that we need to support, or if we can make it more generic somehow.
I think that a call to GroupingOptions.Builder#numReducers is needed to set the number of reducers to 1 until CRUNCH-51 is taken care of (that's kind of on the backburner at the moment, but I am still working on it).
More info in the javadocs about what the class does would be good. The SecondarySort example explains it pretty well, but it's not directly clear how to use SecondarySort when you look at the library class.
It might also be good to change the class name of the SecondarySort example (as it stands, there are two SecondarySort classes). Same goes for the input files for secondary_sort_input.txt file.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477988#comment-13477988 ]
Matthias Friedrich commented on CRUNCH-96:
------------------------------------------
Would now be the time to move the sorting stuff to a o.a.c.lib.sort package? If you expect more additions here it might be best to get it over with.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477311#comment-13477311 ]
Josh Wills commented on CRUNCH-96:
----------------------------------
[~gabriel.reid] please take a look at this one when you get a chance.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-96:
-----------------------------
Attachment: CRUNCH-96.patch
A patch for the secondary sort functionality.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-96:
-----------------------------
Attachment: CRUNCH-96b.patch
Latest and greatest based on Gabriel's feedback-- sorry for the delay, stuff is crazy pre-Hadoop World next week.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96b.patch, CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478301#comment-13478301 ]
Gabriel Reid commented on CRUNCH-96:
------------------------------------
Just one other point as well -- now that you explained the type of use cases that this would be used for, it seems super clear and obvious to me (even though it wasn't when I was just looking at the code), so it might be a good idea to even give a bit of info about the class of use cases that can be solved in the javadocs.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-96) Add secondary sort functionality to
o.a.c.lib
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills resolved CRUNCH-96.
------------------------------
Resolution: Fixed
Committed.
> Add secondary sort functionality to o.a.c.lib
> ---------------------------------------------
>
> Key: CRUNCH-96
> URL: https://issues.apache.org/jira/browse/CRUNCH-96
> Project: Crunch
> Issue Type: Improvement
> Components: Core, MapReduce Patterns
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-96b.patch, CRUNCH-96.patch
>
>
> I've been working on a problem that required a secondary sorting pattern that was very similar to the example that Alex Kozlov created in CRUNCH-78, so it would be good to extract the pattern from the example and move it to o.a.c.lib so it can be easily available to clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira