You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org> on 2012/09/05 23:01:11 UTC
[jira] [Created] (CRUNCH-57) Add a length function to PCollection
Kiyan Ahmadizadeh created CRUNCH-57:
---------------------------------------
Summary: Add a length function to PCollection
Key: CRUNCH-57
URL: https://issues.apache.org/jira/browse/CRUNCH-57
Project: Crunch
Issue Type: New Feature
Components: Core
Affects Versions: 0.3.0
Reporter: Kiyan Ahmadizadeh
Assignee: Josh Wills
Sometimes it's useful and interesting to compute the number of elements in a PCollection.
For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449159#comment-13449159 ]
Josh Wills commented on CRUNCH-57:
----------------------------------
My only quibble is that we force the materialization of the length() when the method is called, which is different than most of our other methods which don't actually run the MR and return the result until the client requests that we read it. Do we want something here like a Future or a Scala option instance?
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450368#comment-13450368 ]
Rahul Sharma commented on CRUNCH-57:
------------------------------------
I think I am missing a point, every PCollection is written in form of Writable/AvroRecord on MapReduce. So that means the data that it written on MR has comparable implemented in it.Then why remove the min/max/sort APIs from PCollection. I addition to this we use a comparator in the min/max API to determine the result. Why force the user to have T implements comparator, we could convert this data into Writable and then do a comparison ?
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450092#comment-13450092 ]
Josh Wills commented on CRUNCH-57:
----------------------------------
+1 to Kiyan.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-57) Add a length function to PCollection
Posted by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kiyan Ahmadizadeh updated CRUNCH-57:
------------------------------------
Attachment: CRUNCH-57.patch
New revision that uses PObject.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449678#comment-13449678 ]
Josh Wills commented on CRUNCH-57:
----------------------------------
I'm +1 for PObject and I like Gabriel's approach re: domain-specific fluent wrappers. At the goog, the FJ PCollection/PTable interfaces were *huge*, but it made writing fluent pipelines really easy. My initial idea was to put all of the MR patterns into the lib.* module, but it ended up being annoying to work with in practice, IMO.
Kiyan, do you have an opinion on how you want to go about this one? Do you want to take on defining PObject (which in my mind, could just be a simple wrapper that materialized a PCollection and then implemented some abstract function that did a computation on the materialized Iterable) and incorporate it here? Or do you just want to keep the Aggregate.length impl for now and leave off the rest of the change until we create PObject?
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-57) Add a length function to PCollection
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-57:
-------------------------------
Attachment: MinMaxFn.patch
Patch with implementations of min and max function. The function do not rely on S being comparable instead they work on the corresponding MapReduce WritableKey type.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449195#comment-13449195 ]
Josh Wills commented on CRUNCH-57:
----------------------------------
+1 from me.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455630#comment-13455630 ]
Gabriel Reid commented on CRUNCH-57:
------------------------------------
@Rahul, about the MinMaxFn.patch -- I think that this will cause problems if a non-writable class (for example, with a custom compare implementation) is used with a different backing writable class.
Additionally, it won't give a very clear error when the value really isn't comparable, like in the case of a custom writable class that isn't comparable, or a avro-reflection class.
In general I think we want to stay away from basing things on the underlying serialization scheme, as the underlying serialization may have different compare logic (either implicitly or explicitly) than the type of object in the PCollection
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451256#comment-13451256 ]
Gabriel Reid commented on CRUNCH-57:
------------------------------------
@Rahul, I think you're confusing the use of sorting with min/max calculations. Min and max aren't calculated using sort, they're calculated by doing an initial filter in the mappers and then a final filter in a second reducer.
This approach will always be more efficient than sorting the PCollection first. Outside of the MR context, calculating min/max performance is O(n), and sort is O(n log n) in the average case -- this translates into resource usage in MR, although of course things are done in parallel.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450059#comment-13450059 ]
Matthias Friedrich commented on CRUNCH-57:
------------------------------------------
Adding length() to PCollection and removing min(), max(), and sort() instead makes perfect sense to me. How would we do that? Deprecate for a couple of releases or remove them right away?
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450084#comment-13450084 ]
Kiyan Ahmadizadeh commented on CRUNCH-57:
-----------------------------------------
+1 On adding length() to PCollection and removing min(), max(), and source() into a decorator. Deprecation then removing in a couple of releases seems like the standard way to go.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455657#comment-13455657 ]
Gabriel Reid commented on CRUNCH-57:
------------------------------------
@Rahul, I don't really think that this is a situation that we need to overly optimize for right now, and I definitely wouldn't say that we're losing what Hadoop can do for us out of the box. I think that CPU usage will only be slightly lowered by making use of WritableComparable, as the main objective here is to minimize the amount of data being sent to the reducer. Additionally, the current implementation makes use of a Combiner, so it's likely that it'll be even more successful in minimizing the size of the shuffle.
In any case, I think we're going pretty far outside the scope of this particular JIRA issue, so it's probably best to continue this discussion on crunch-dev or a different JIRA issue.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450060#comment-13450060 ]
Josh Wills commented on CRUNCH-57:
----------------------------------
Kiyan makes a good point re: Comparable. I have been hoisted by my own petard. ;-)
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449999#comment-13449999 ]
Kiyan Ahmadizadeh commented on CRUNCH-57:
-----------------------------------------
I'm up for taking on an implementation of PObject and incorporating it into this change. I've created a ticket CRUNCH-58 for this. Josh, please check that ticket for some discussion on the implementation of PObject.
+1 For using decorators to achieve the Fluent pattern without crowding the methods in the PCollection interface. This would work well in Java and Scala. I think Gabriel's geometry example highlights the issue that you may want special operations on PCollections holding objects of a specific type. Another example would be PCollections of numeric data. It would make sense for such collections to have special operations like average, sum, etc.
-1 On not including length() in the base PCollection interface, however. I think decorators are great for the case outlined above, where the functionality applies only to PCollections holding objects of a specific type. Counting the number of elements in a PCollection, however, is applicable to all PCollection regardless of the type of object it contains. I think operations that can apply to any and all PCollections belong in the PCollection interface, and operations applicable to a specific kind of PCollection belong in decorators. For this reason I argue that length() goes in the PCollection interface.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450003#comment-13450003 ]
Kiyan Ahmadizadeh commented on CRUNCH-57:
-----------------------------------------
On a similar note, is it expected that for a PCollection<T>, T is comparable? Aggregate.max and Aggregate.min contain implementations that check that the type class for the PCollection is Comparable. Given the MapReduce framework's reliance on sorting, however, it seems like T should always be Comparable.
If it's not expected that T be Comparable, I would argue that max, min, and sort functionality should be in a PCollection decorator that contains operations for PCollections that contain comparable elements.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills resolved CRUNCH-57.
------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Committed Kiyan's new length() impl that makes use of PObject, and I'm expecting that we'll pick up the min/max/sort thread on another JIRA or on crunch-dev.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Fix For: 0.4.0
>
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450566#comment-13450566 ]
Josh Wills commented on CRUNCH-57:
----------------------------------
Writable isn't a sub-interface of Comparable.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455668#comment-13455668 ]
Rahul Sharma commented on CRUNCH-57:
------------------------------------
My apologies for dragging it so far, but my point was to make the three APIs viz min/max/sort all based on similar concept i.e. either using S as comparable or using WritableComparable and not about any optimizations. I think that the min/max are special cases of sort. But for now, it is best to park this discussion and take it on dev list at some point in time.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449507#comment-13449507 ]
Gabriel Reid commented on CRUNCH-57:
------------------------------------
Clarification on my comment -- I'm in favour of implementing PObject, but like Matthias, I'm not crazy about the idea of adding more methods to PCollection (I have the feeling there are a few too many already). However, the advantage of having lots of methods on PCollection makes it easy to use the Fluent [1] interface pattern.
Going off on a tangent that's probably slightly outside the scope of this issue, we could move towards grouping specialized functionality in PCollection decorators that return instances of their own type (an extension of PCollection), which could allow us to do something like this:
PCollection<Geometry> geomCollection = ...;
PCollection<Geometry> manipulatedGeom = new GeometryOpPCollection(geomCollection)
.someGeometricOperation() // Treating this as a GeometryOpPCollection
.anotherGeometricOperation() // Again treating this as a GeometryOpPCollection
.parallelDo(new StandardFunction()); // But here it's just like a normal PCollection
This way we can continue to support the Fluent interface use case, while not having the PCollection interface explode with new methods.
[1] http://en.wikipedia.org/wiki/Fluent_interface
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449193#comment-13449193 ]
Kiyan Ahmadizadeh commented on CRUNCH-57:
-----------------------------------------
Taking a look at the FlumeJava paper (this copy: http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf), it looks like the answer to this is the PObject, which acts a bit like a Future, although PObjects defer the start of computation until the object is accessed, while a Java Future begins computation as soon as the Future is constructed and blocks on a call to get() if the Future has yet to complete.
It seems like the PObject concept would be generally useful. Min and max on PCollection could be changed to return a PObject, as could this length method, etc.
Perhaps we should make another ticket for implementing PObject. Thoughts?
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-57) Add a length function to PCollection
Posted by "Kiyan Ahmadizadeh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kiyan Ahmadizadeh updated CRUNCH-57:
------------------------------------
Attachment: CRUNCH-57.patch
This patch gives an implementation. A static length() method is added to Aggregate.java and used from PCollectionImpl and MemCollection to implement a length() function defined in the PCollection interface. The implementation for Aggregate#length closely follows the implementation of Aggregate#count with a few tweaks.
This functionality is exposed in Scala by adding a length function to trait PCollectionLike.
Integration tests are added for this functionality.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-57) Add a length function to PCollection
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-57:
-------------------------------
Attachment: minver2.patch
@Gabriel, yes you are right that the approach outside MR context will be faster, but in MR we have few things that come into play like eg when we are using a reducer after groupByKey then MR will put sorting in place, if we use it or not that's secondary but on reducer the output will be sorted always.
I have created a version for min function that tries to use things from MR and following the same principle. I tested it against the avro data in aggregate test. It is a bit faster that the current min function like the best result clocked 9% faster and in worst result it was the same. Another important aspect is it doesn't rely on user classes being comparable.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451248#comment-13451248 ]
Rahul Sharma commented on CRUNCH-57:
------------------------------------
If I look at sort, then we are putting data on MR in form of Writable-Key, Void-value. Now any data that is written in form of key on MR is supposed to be WritableComparable, This means that the converted format would be comparable. The point here is the writable type is comparable and not the type T. If we try to do it the other way i.e not making the writable as comparable and relying on T being comparable I do not think that we would be able to tap the power hadoop, as the data could not be written as key in hadoop and thus could not be sorted. This way could turn out to be highly inefficient.
In the aggregate funcs of min/man currently we are putting data in form of Boolean-key/writable-value format.Now all keys values are false here. But we could implement the min/max APIs in the a way where we put hadoop sorting to some use. I am not sure but I believe this could be a bit more efficient than the current one. This would mean all the types would have a comparable.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455645#comment-13455645 ]
Rahul Sharma commented on CRUNCH-57:
------------------------------------
@Gabriel Yes you are right on both the issues. We can address them by asking for a Comparable S. But if we make things WritableComparable then we could tap hadoop.
Also the Sort API suffers from both these issues. There also if the Writable is not comparable then the error will not be that clear and S could have a comparator that is different from the Writable. I would think that we would like to have a similar fix here i.e. basing things on S. If we try to make Comparable fixes here then we will loose what hadoop could do for us, out of the box. If we can keep Sort API like that why not let the min/max funcs also. Any thoughts ? or Can we do some things in Sort API to make it in sync with min/max ?
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch, CRUNCH-57.patch, MinMaxFn.patch, minver2.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449388#comment-13449388 ]
Gabriel Reid commented on CRUNCH-57:
------------------------------------
+1 for adding an issue to implement PObject -- that would be very handy, and definitely seems like the right way to give access to high-level aggregation like a count of elements.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-57) Add a length function to PCollection
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449454#comment-13449454 ]
Matthias Friedrich commented on CRUNCH-57:
------------------------------------------
-0 from me. Do we really want to add even more methods to PCollection? Shouldn't our core abstractions be minimal? An interface with more than 20 methods looks to me like it's about to lose focus. I think it would suffice having length() in Aggregation.
> Add a length function to PCollection
> ------------------------------------
>
> Key: CRUNCH-57
> URL: https://issues.apache.org/jira/browse/CRUNCH-57
> Project: Crunch
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.3.0
> Reporter: Kiyan Ahmadizadeh
> Assignee: Josh Wills
> Attachments: CRUNCH-57.patch
>
>
> Sometimes it's useful and interesting to compute the number of elements in a PCollection.
>
> For example, suppose there was an initial PCollection that was then filtered into another. If I'm interested in how many elements of the original PCollection matched the filter, I'll have to write extra code to compute this.
> PCollections should have a length method that, when called, computes the number of elements in the PCollection and returns the result.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira