You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2012/07/26 07:59:33 UTC
[jira] [Created] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Josh Wills created CRUNCH-26:
--------------------------------
Summary: Make Scrunch tests suitable for continuous integration
Key: CRUNCH-26
URL: https://issues.apache.org/jira/browse/CRUNCH-26
Project: Crunch
Issue Type: Task
Components: Scrunch
Affects Versions: 0.3.0
Reporter: Josh Wills
The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424344#comment-13424344 ]
Matthias Friedrich commented on CRUNCH-26:
------------------------------------------
That's not exactly what I had in mind with crunch-test but it'll work, so let's do it. I'll add add patch for this later today.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch, CRUNCH-26b.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills reassigned CRUNCH-26:
--------------------------------
Assignee: Josh Wills
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424019#comment-13424019 ]
Matthias Friedrich commented on CRUNCH-26:
------------------------------------------
Looks good, I just noticed one issue: You had to depend on Crunch's test-jar to be able to use TemporaryPath. The downside is that this breaks "mvn install -Dmaven.test.skip=true" unless you can get the test-jar from some repository. I see two options here: a) don't use TemporaryPath, instead implement something in Scala that fits better, or b) create a new module crunch-test where we put TemporaryPath and other testing helpers that we or our users might need.
If you decide for b) I wouldn't mind doing it.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424040#comment-13424040 ]
Josh Wills commented on CRUNCH-26:
----------------------------------
It seems like b is the right option long-term, as I expect there will be other subprojects that get created over time that will want to use the TemporaryPath functionality.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424156#comment-13424156 ]
Josh Wills commented on CRUNCH-26:
----------------------------------
But the TemporaryPath dependency is the only one needed by Scrunch, and I expect that any others that were needed across multiple projects could be refactored so as to minimize their crunch-core dependencies, just as we can w/TemporaryPath. Passing the value of the RuntimeParameters.TMP_DIR as a constructor argument isn't much of a hardship for users of TemporaryPath.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424114#comment-13424114 ]
Matthias Friedrich commented on CRUNCH-26:
------------------------------------------
After the executing the scrunch test suite I noticed two left-over files: There's still a call to Files.createTempDir() in InterpeterRunner.scala and PipelineAppTest.scala leaves a /tmp/hadoop-$USER dir behind.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills resolved CRUNCH-26.
------------------------------
Resolution: Fixed
Fix Version/s: 0.3.0
Committed with a couple of small fixes-- thanks Matthias!
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.3.0
>
> Attachments: CRUNCH-26.patch, CRUNCH-26_mafr.patch, CRUNCH-26b.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-26:
-----------------------------
Attachment: CRUNCH-26.patch
A patch to make Scrunch use the TemporaryPath code and remove all FileHelper references.
This required a couple of hacks, since Scala doesn't really have such a thing as public member variables, and the @Rule annotation requires them. I got this to work by defining a base java class for all of the scrunch IT tests, and having that base Java class define the @Rule-annotated TemporaryPath instance that the scrunch tests used. Not the prettiest thing, but it works for now, and we'll be able to clean it up a bit when JUnit 4.11 is released.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias Friedrich updated CRUNCH-26:
-------------------------------------
Attachment: CRUNCH-26_mafr.patch
Here comes the promised crunch-test change :)
Apologies for the huge patch, but changes are mostly mechanic so it should still be easy to review. I used Josh's getDefaultConfiguration() method while I was at it because it's easier on the eye.
Test suite output in /tmp is now pretty minimal, there's not much we can do about the bug in maven-scala-plugin though.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch, CRUNCH-26_mafr.patch, CRUNCH-26b.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424111#comment-13424111 ]
Matthias Friedrich commented on CRUNCH-26:
------------------------------------------
Hmm, now I realize this doesn't work. The new crunch-test would have to depend on crunch-core to offer useful functionality. But since the integration tests in crunch-core need crunch-test, we'd have a cyclic dependency (which Maven would probably prevent). The only way out of this would be to move integration tests from crunch-core into a separate Maven module. But that's a rather drastic change and I think we decided against this before.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424126#comment-13424126 ]
Josh Wills commented on CRUNCH-26:
----------------------------------
Wait-- why would crunch-test need to depend on crunch-core?
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424151#comment-13424151 ]
Matthias Friedrich commented on CRUNCH-26:
------------------------------------------
Right now it only depends on RuntimeParameters for a constant but in the future there might be more. I think spring-test would we a good analogy; it depends on spring core libraries and offers mock objects and other functionality for building integration tests.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-26) Make Scrunch tests suitable for
continuous integration
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-26:
-----------------------------
Attachment: CRUNCH-26b.patch
Fix the PipelineAppTest and the InterpreterRunnerTest to use the junit temp directories.
> Make Scrunch tests suitable for continuous integration
> ------------------------------------------------------
>
> Key: CRUNCH-26
> URL: https://issues.apache.org/jira/browse/CRUNCH-26
> Project: Crunch
> Issue Type: Task
> Components: Scrunch
> Affects Versions: 0.3.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-26.patch, CRUNCH-26b.patch
>
>
> The scrunch equivalent of the changes done to Crunch in CRUNCH-24.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira