You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Thomas Weise (JIRA)" <ji...@apache.org> on 2012/05/21 06:08:41 UTC
[jira] [Created] (PIG-2711) e2e harness: cache benchmark results
between test runs
Thomas Weise created PIG-2711:
---------------------------------
Summary: e2e harness: cache benchmark results between test runs
Key: PIG-2711
URL: https://issues.apache.org/jira/browse/PIG-2711
Project: Pig
Issue Type: Improvement
Components: e2e harness
Reporter: Thomas Weise
Assignee: Thomas Weise
It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2711:
----------------------------
Attachment: PIG-2711-2.patch
Yes, it mean to be optional. Attach the patch for fix. Can you try it Jie?
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711-2.patch, PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Thomas Weise (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288793#comment-13288793 ]
Thomas Weise commented on PIG-2711:
-----------------------------------
Also, this was meant to be optional (see 0.10 branch). The build.xml in trunk may have additional modifications that don't set PH_BENCHMARK_CACHE_PATH empty by default.
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2711.
-----------------------------
Resolution: Fixed
Tested. Commit PIG-2711-2.patch to 0.10/trunk.
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711-2.patch, PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Thomas Weise (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated PIG-2711:
------------------------------
Attachment: (was: PIG-2711.patch)
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2711:
----------------------------
Fix Version/s: 0.10.1
Also commit to 0.10 branch as per Thomas' request.
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Thomas Weise (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated PIG-2711:
------------------------------
Attachment: PIG-2711.patch
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288808#comment-13288808 ]
Jie Li commented on PIG-2711:
-----------------------------
Thanks Thomas! That works. How about adding this to https://cwiki.apache.org/confluence/display/PIG/HowToTest?
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Thomas Weise (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279952#comment-13279952 ]
Thomas Weise commented on PIG-2711:
-----------------------------------
For one this will speed up subsequent runs as we eliminate redundant pig runs (on some clusters running the harness at present takes up to 24 hours with local and mapred mode and additional internal tests).
This feature can also be used to compare pig and hadoop versions based on archived benchmarks. While the harness currently allows to configure old and new pig and hadoop versions, it would be much simpler to configure the benchmark than recomputing it.
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2711.
-----------------------------
Resolution: Fixed
Fix Version/s: 0.11
Hadoop Flags: Reviewed
Patch committed to trunk. Thanks Thomas!
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11
>
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li reopened PIG-2711:
-------------------------
Do we need to update the documentation for setting up the e2e environment? I couldn't get it work due to the following error:
(MR test job running succesfully...)
ERROR TestDriver::run at : 470 Failed to run test FilterBoolean_1 <Unable to open file ${PH_BENCHMARK_CACHE_PATH}/FilterBoolean_1_benchmark.pig to write pig script, No such file or directory
>
Even I passed -DPH_BENCHMARK_CACHE_PATH=some_path, I still got the error:
ERROR TestDriver::run at : 470 Failed to run test FilterBoolean_1 <Unable to open file /Users/JieLi/svn/pig/pig_benchmark_cache/FilterBoolean_1_benchmark.pig to write pig script, No such file or directory
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289837#comment-13289837 ]
Jie Li commented on PIG-2711:
-----------------------------
Sure let me try it:)
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711-2.patch, PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2711) e2e harness: cache benchmark results
between test runs
Posted by "Thomas Weise (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288790#comment-13288790 ]
Thomas Weise commented on PIG-2711:
-----------------------------------
What you specify needs to be an existing path. Did you create /Users/JieLi/svn/pig/pig_benchmark_cache ?
> e2e harness: cache benchmark results between test runs
> ------------------------------------------------------
>
> Key: PIG-2711
> URL: https://issues.apache.org/jira/browse/PIG-2711
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.11, 0.10.1
>
> Attachments: PIG-2711.patch
>
>
> It will be helpful to be able to store the benchmark results that are computed with "old pig" and reuse in subsequent runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira