You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/01/06 07:11:34 UTC
[jira] Created: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
[hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
----------------------------------------------------------------------------------
Key: HADOOP-2533
URL: https://issues.apache.org/jira/browse/HADOOP-2533
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Reporter: stack
Priority: Minor
Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Status: In Progress (was: Patch Available)
Hudson queue is empty. Try requeuing to see if that'll get an entry in the queue.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533-v3.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Attachment: dirty.patch
Attached dirty patch -- it needs a bunch of work to make it committable -- shrinks MapWritable CPU usage over a PerformanceEvaluation run from 22.9% of all CPU used down to 0.8%. It uses a dumber MapWritable -- one that has static maps of classes to codes -- and one that is more friendly to hbase; it has been primed with hbase type (they have to be dynamically 'added' to each new instantiation of the hadoop.io version of MapWritable).
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Attachments: dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Attachment: 2533.patch
Cleaned up patch
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Attachments: 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Status: Patch Available (was: In Progress)
Retry submit to see if resubmit gets it on to the queue
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533-v3.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Status: In Progress (was: Patch Available)
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Fix Version/s: 0.16.0
Status: Patch Available (was: Open)
Passes tests locally. Passing it to hudson.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Attachment: 2533-v3.patch
TestScanner2 has been renamed so patch wouldn't apply.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533-v3.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558365#action_12558365 ]
Hadoop QA commented on HADOOP-2533:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373037/2533-v3.patch
against trunk revision r611537.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1567/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1567/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1567/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1567/console
This message is automatically generated.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533-v3.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Attachment: 2533-v2.patch
Patch wouldn't apply (hbase had changed meantime). Add up v2 version of patch that applies to TRUNK
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557485#action_12557485 ]
Hadoop QA commented on HADOOP-2533:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372658/2533.patch
against trunk revision r610603.
@author +1. The patch does not contain any @author tags.
patch -1. The patch command could not apply the patch.
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1531/console
This message is automatically generated.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HADOOP-2533:
-----------------------------
Assignee: stack
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558170#action_12558170 ]
Hadoop QA commented on HADOOP-2533:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372851/2533-v2.patch
against trunk revision r611333.
@author +1. The patch does not contain any @author tags.
patch -1. The patch command could not apply the patch.
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1549/console
This message is automatically generated.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558596#action_12558596 ]
Hudson commented on HADOOP-2533:
--------------------------------
Integrated in Hadoop-Nightly #365 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/365/])
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533-v3.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Status: Patch Available (was: In Progress)
Trying hudson again
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2533) [hbase] Performance: Scanning, just
creating MapWritable in next consumes >20% CPU
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2533:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed (a while back). Resolving.
> [hbase] Performance: Scanning, just creating MapWritable in next consumes >20% CPU
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2533
> URL: https://issues.apache.org/jira/browse/HADOOP-2533
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: 2533-v2.patch, 2533-v3.patch, 2533.patch, dirty.patch
>
>
> Every call to HScanner.next creates an instance of MapWritable. MapWritables are expensive. Watching a scan run in the profiler, the setup of the MapWritable -- filling out the idToClassMap and classToIdMap -- consumes 20% of all CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.