You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Brian Martin (JIRA)" <ji...@apache.org> on 2012/08/02 02:22:02 UTC
[jira] [Created] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Brian Martin created CRUNCH-30:
----------------------------------
Summary: Add cross join to Scrunch PTable and PCollection
Key: CRUNCH-30
URL: https://issues.apache.org/jira/browse/CRUNCH-30
Project: Crunch
Issue Type: Improvement
Components: Scrunch
Affects Versions: 0.3.0
Reporter: Brian Martin
Fix For: 0.3.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Brian Martin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Martin updated CRUNCH-30:
-------------------------------
Attachment: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch, 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427048#comment-13427048 ]
Josh Wills commented on CRUNCH-30:
----------------------------------
Run mvn verify to kick off the integration tests.
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Brian Martin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430514#comment-13430514 ]
Brian Martin commented on CRUNCH-30:
------------------------------------
Thanks! target/rat.txt is what I was missing.
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch, 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429960#comment-13429960 ]
Matthias Friedrich commented on CRUNCH-30:
------------------------------------------
Brian, the rat plugin checks for files that don't have Apache license headers. Check the rat.txt files in the target directories to see which file it is. In most cases it's your own patch in the project's base directory :)
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch, 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Brian Martin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429560#comment-13429560 ]
Brian Martin commented on CRUNCH-30:
------------------------------------
Thanks for the comments Josh. I've attached a newer patch that addresses those issues.
Potential problems:
- scrunch/src/test/scala is not listed anywhere in the pom.xml. will this be included on the buildpath automatically?
- I couldn't run the test via mvn test due to rat-check problems, but ran the test from eclipse.
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch, 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Brian Martin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Martin updated CRUNCH-30:
-------------------------------
Attachment: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
I'm a little confused about how testing works now. I ran the included test via: mvn -Dtest=CrossJoinTest test, but just running "mvn test" doesn't appear to run any tests for me.
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills resolved CRUNCH-30.
------------------------------
Resolution: Fixed
Committed-- thanks Brian!
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch, 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-30) Add cross join to Scrunch PTable and
PCollection
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428938#comment-13428938 ]
Josh Wills commented on CRUNCH-30:
----------------------------------
Thanks for this-- three things I noticed when I ran it locally:
1) Whitespace issues in the CrossJoinTest-- some lines have tabs for some reason.
2) The CrossJoinTest isn't really an integration test-- it runs entirely in-memory, and so it should live in src/test/scala instead of src/it/scala.
3) I think that the toMap call in the materializeToMap function would trigger a MapReduce pipeline to run by pulling the data from the materialized instance when it is called-- inserting a .view call in there, or delegating to the materialize() method in PTable that does that already, is a better choice.
> Add cross join to Scrunch PTable and PCollection
> ------------------------------------------------
>
> Key: CRUNCH-30
> URL: https://issues.apache.org/jira/browse/CRUNCH-30
> Project: Crunch
> Issue Type: Improvement
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Brian Martin
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-30-Add-cross-join-to-Scrunch-PTable-and-PColl.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira