You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Rahul Sharma (JIRA)" <ji...@apache.org> on 2012/09/24 13:28:07 UTC
[jira] [Created] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Rahul Sharma created CRUNCH-75:
----------------------------------
Summary: Have a method to create Bloom filters in Crunch
Key: CRUNCH-75
URL: https://issues.apache.org/jira/browse/CRUNCH-75
Project: Crunch
Issue Type: New Feature
Reporter: Rahul Sharma
As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462430#comment-13462430 ]
Rahul Sharma commented on CRUNCH-75:
------------------------------------
There was a discussion for BloomFilters, to have them at a place that is something on lines of PiggyBank. And my apologies that I didn't initiate a thread to discus the module but I created one that I had in mind. I will initiate a thread for the same .
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-75:
-------------------------------
Attachment: 0001-CRUNCH-75.patch
Patch updated, using CrunchTestSupport. My bad I should have used that earlier.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0001-CRUNCH-75.patch, 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma resolved CRUNCH-75.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Committed.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Fix For: 0.4.0
>
> Attachments: 0001-CRUNCH-75.patch, 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-75:
-------------------------------
Attachment: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch
Created a module crunch-bytes which can have user functions.
Added methods to create BloomFilters for files at a location and for a PCollection. Filter creation will return back PObject types.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471601#comment-13471601 ]
Josh Wills commented on CRUNCH-75:
----------------------------------
Not a problem-- +1, this is a nice addition.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0001-CRUNCH-75.patch, 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma reassigned CRUNCH-75:
----------------------------------
Assignee: Rahul Sharma
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462113#comment-13462113 ]
Josh Wills commented on CRUNCH-75:
----------------------------------
Very cool. Q: was the intent for crunch-bytes to be a general purpose repo, ala piggybank? If so, how did we come up with the name? I don't remember the context.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471241#comment-13471241 ]
Matthias Friedrich commented on CRUNCH-75:
------------------------------------------
Other than the issue Josh pointed out this looks good now, thank you!
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-75:
-------------------------------
Attachment: CRUNCH-75.patch
Updated the patch for changes.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469117#comment-13469117 ]
Rahul Sharma commented on CRUNCH-75:
------------------------------------
Matthias +1 for the suggested changes, I will update the javadocs and other required ones. As for the compilation issue, the patch depends on CRUNCH-74, so you need to apply that patch and then it will start building.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468629#comment-13468629 ]
Rahul Sharma edited comment on CRUNCH-75 at 10/4/12 2:55 AM:
-------------------------------------------------------------
Made crunch-contrib as a dependency of crunch-dist. I tried doing git diff --find-hard-copies, but I do not think that it made much difference. Please let me know if something else was required, I will update the patch.
was (Author: rahul.sharma):
Made crunch-contrib as a dependency of crunch-dist. I tried doing git diff --find-hard-copies, but I do not think that it made much difference. I am not sure if something else was required.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471104#comment-13471104 ]
Josh Wills commented on CRUNCH-75:
----------------------------------
@Rahul you should use CrunchTestSupport and not include TemporaryPaths.java in crunch-test. We don't want to introduce a recursive dependency between crunch and crunch-test by referencing RuntimeParameters in crunch-test. CrunchTestSupport does what you want TemporaryPaths to do, just have your IT.java classes extend it. Otherwise, +1.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-75:
-------------------------------
Attachment: 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch
Added BloomFilters to crunch-contrib
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467644#comment-13467644 ]
Matthias Friedrich commented on CRUNCH-75:
------------------------------------------
Two things I noticed at first glance: You'll have to add crunch-contrib as a dependency of crunch-dest, otherwise it won't be part of the binary release (see CRUNCH-76). Second thing: Could you create the patch via --find-copies-harder, that'll make it much shorter and easier to read.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-75) Have a method to create Bloom
filters in Crunch
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468650#comment-13468650 ]
Matthias Friedrich commented on CRUNCH-75:
------------------------------------------
Hi Rahul! I meant "git format-patch --find-copies-harder master". I tried that locally and found the problem: Our shakes.txt has CRLF line endings while the one you add has LF line endings (which is correct). Without --find-copies-harder, the patch just adds the file, with --find-copies-harder, the patch makes a copy and then changes line endings on every single line. So that's bad luck, sorry about the confusion!
As for the patch: Right now it doesn't compile for me, and I'd also put everything in crunch-contrib into the Java package "org.apache.crunch.contrib" so it's grouped nicely when we create aggregated Javadoc. Speaking of Javadoc, could you add some, along with a package-info.java file for o.a.c.contrib and o.a.c.contrib.bloomfilter?
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-75) Have a method to create Bloom filters
in Crunch
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Sharma updated CRUNCH-75:
-------------------------------
Attachment: CRUNCH-75.patch
Made crunch-contrib as a dependency of crunch-dist. I tried doing git diff --find-hard-copies, but I do not think that it made much difference. I am not sure if something else was required.
> Have a method to create Bloom filters in Crunch
> -----------------------------------------------
>
> Key: CRUNCH-75
> URL: https://issues.apache.org/jira/browse/CRUNCH-75
> Project: Crunch
> Issue Type: New Feature
> Reporter: Rahul Sharma
> Assignee: Rahul Sharma
> Attachments: 0002-Created-crunch-bytes-module-for-users-to-share-funct.patch, 0002-CRUNCH-75-Added-BloomFilters-to-crunch-contrib.patch, CRUNCH-75.patch
>
>
> As per discussion on mailing list, Bloomfilters provide interesting addon to crunch. But the crunch-core is not the correct place to add such a thing. So create a module i.e. something along the lines on piggybank of Pig.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira