You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org> on 2011/04/22 04:17:05 UTC
[jira] [Created] (PIG-2011) Speed up TestTypedMap.java
Speed up TestTypedMap.java
---------------------------
Key: PIG-2011
URL: https://issues.apache.org/jira/browse/PIG-2011
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Assignee: Dmitriy V. Ryaboy
Fix For: 0.10
TestTypedMap uses Mapreduce mode and takes 7 minutes.
This can be improved.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2011) Speed up TestTypedMap.java
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023304#comment-13023304 ]
Daniel Dai commented on PIG-2011:
---------------------------------
+1. I agree we can convert a large amount of tests to local mode, except for some order by/skewed join tests, which will behave differently in local mode.
> Speed up TestTypedMap.java
> ---------------------------
>
> Key: PIG-2011
> URL: https://issues.apache.org/jira/browse/PIG-2011
> Project: Pig
> Issue Type: Improvement
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.10
>
> Attachments: PIG_2011.patch
>
>
> TestTypedMap uses Mapreduce mode and takes 7 minutes.
> This can be improved.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2011) Speed up TestTypedMap.java
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2011:
-----------------------------------
Status: Patch Available (was: Open)
> Speed up TestTypedMap.java
> ---------------------------
>
> Key: PIG-2011
> URL: https://issues.apache.org/jira/browse/PIG-2011
> Project: Pig
> Issue Type: Improvement
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.10
>
> Attachments: PIG_2011.patch
>
>
> TestTypedMap uses Mapreduce mode and takes 7 minutes.
> This can be improved.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2011) Speed up TestTypedMap.java
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2011:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk (0.10).
> Speed up TestTypedMap.java
> ---------------------------
>
> Key: PIG-2011
> URL: https://issues.apache.org/jira/browse/PIG-2011
> Project: Pig
> Issue Type: Improvement
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.10
>
> Attachments: PIG_2011.patch
>
>
> TestTypedMap uses Mapreduce mode and takes 7 minutes.
> This can be improved.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2011) Speed up TestTypedMap.java
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2011:
-----------------------------------
Attachment: PIG_2011.patch
Attached patch switches TestTypedMap to using LOCAL mode. This results in pretty significant savings:
IN MR:
Testcase: testSimpleLoad took 25.21 sec
Testcase: testSimpleMapKeyLookup took 23.7 sec
Testcase: testSimpleMapCast took 23.432 sec
Testcase: testComplexLoad took 23.539 sec
Testcase: testComplexCast took 19.355 sec
Testcase: testComplexCast2 took 18.456 sec
Testcase: testUnTypedMap took 18.373 sec
Testcase: testOrderBy took 171.013 sec
Total time: 7 min 10 secs
IN LOCAL:
Testcase: testSimpleLoad took 8.758 sec
Testcase: testSimpleMapKeyLookup took 8.947 sec
Testcase: testSimpleMapCast took 8.781 sec
Testcase: testComplexLoad took 8.072 sec
Testcase: testComplexCast took 7.843 sec
Testcase: testComplexCast2 took 7.787 sec
Testcase: testUnTypedMap took 8.595 sec
Testcase: testOrderBy took 47.002 sec
Total time: 2 min 33 secs
> Speed up TestTypedMap.java
> ---------------------------
>
> Key: PIG-2011
> URL: https://issues.apache.org/jira/browse/PIG-2011
> Project: Pig
> Issue Type: Improvement
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.10
>
> Attachments: PIG_2011.patch
>
>
> TestTypedMap uses Mapreduce mode and takes 7 minutes.
> This can be improved.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira