You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Clint Morgan (JIRA)" <ji...@apache.org> on 2007/11/06 18:34:51 UTC
[jira] Created: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
getRow() is orders of magnitudes slower than get(), even on rows with one column
--------------------------------------------------------------------------------
Key: HADOOP-2161
URL: https://issues.apache.org/jira/browse/HADOOP-2161
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Affects Versions: 0.16.0
Environment: latest from trunk
Reporter: Clint Morgan
HTable.getRow(Text) is several orders of magnitude slower than
HTable.get(Text, Text), even on rows with a single column.
This problem can be observed by the attached patch of
PerformanceEvaluation.java which changes SequentialRead to use getRow,
and prints out the time for each read.
The test can the be run with:
bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
On my laptop, the original test (using get()) produces reads on the order of 5-20
milliseconds. Using getRow(), the reads take 50-2000 ms.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541119 ]
stack commented on HADOOP-2161:
-------------------------------
Looks like memcache has the same issue. Let me write a test...
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541538 ]
Hudson commented on HADOOP-2161:
--------------------------------
Integrated in Hadoop-Nightly #299 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/299/])
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Fix For: 0.16.0
>
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Resolution: Fixed
Fix Version/s: 0.16.0
Status: Resolved (was: Patch Available)
Committed. Thanks for the initial patch Clint.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Fix For: 0.16.0
>
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540661 ]
stack commented on HADOOP-2161:
-------------------------------
Actually, I misspoke. getFull scanning memory and all on-disk files is not 'wrong' -- though it is slow. Here's why.
Columns can be added willy-nilly. There is no need of an ALTER TABLE-like statement adding a column as there is in a traditional RDBMS -- as long as the column belongs to an existing column family (has an extant column family for a prefix).
And there is no accounting anywhere in hbase of all the columns made in any particular family. Since there is no list of all-columns to consult, the only way hbase can be sure its found all column mentions is if it scans all data. This is main difference between get and getFull. Because you provide a list of columns to fetch to get, it can know when its done. Not so with getFull.
Is it important to you that this run faster Clint? If so, there may be some things we can do like keep an integer of counts of unique column names. getFull would know that when it had hit the count of all column names, it could return (Keeping a list of all column names would probably not be viable since in some schemas it might grow without bound).
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541375 ]
Clint Morgan commented on HADOOP-2161:
--------------------------------------
Tried out the patch and it works fine. The memcache fix did take care of the behavior I commented on previously.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Attachment: 2161-3.patch
Remove the Clint patch from PerformanceEvaluation that logged times and used getFull instead of get.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161-3.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540927 ]
stack commented on HADOOP-2161:
-------------------------------
I'm currently looking at this too. One issue I've just found is that rows are lexicographically sorted but rows in PE are numbers (This makes it so 4500 < 46). The MapFiles are messed up. Am going to zero-pad to see what that does (By the way, thanks for digging in here and finding non-breaking after we'd left the row).
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clint Morgan updated HADOOP-2161:
---------------------------------
Attachment: PerformanceEvaluation-patch.txt
Modifies SequentialReadTest to use getRow, and print the read time to standard out.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541404 ]
Hadoop QA commented on HADOOP-2161:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12369212/2161-4.patch
against trunk revision r592860.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1081/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1081/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1081/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1081/console
This message is automatically generated.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Attachment: tg2.patch
TestGet2 that works against TRUNK for BryanD
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Fix For: 0.16.0
>
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt, tg2.patch
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clint Morgan updated HADOOP-2161:
---------------------------------
Attachment: HADOOP-2161-2.patch
Well, 2 sec read times are not acceptable for us. But looking into
this further, there appears to be a bug causing the excessive time
(rather than the need to look everywhere for the row).
Looking at HStore.getFull (line 1103), the break from the while loop
should occur when key.compareTo(readKey) is LESS than zero. This
cuts the read times back down to 5-20 ms.
However, there still seems to be a problem:
When there are two MapFiles in the HStore, (again in getFull()) After
calling map.getClosest(), matching the key and storing the results,
the call to map.next() produces a key that is much less than the key
returned by closest. So time is wasted again iterating through all
theses keys to get back to closest, and out of the loop.
This problem can be observed by applying my patch to HStore, running
my pached performance test, and when the times start to climb tracing
through getFull().
I had quick look at what was going on with getClosest and next, but
did not understand all that was going on.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540922 ]
Clint Morgan commented on HADOOP-2161:
--------------------------------------
I am having trouble getting the problem I raised in the last comment
to occur in the stock PerformanceEvaluation. However, if I drop the
number of rows in the test down an order of magnitude, by changing line 82 to
private static final int ONE_GB = 1024 * 1024 * 100;
I can see the problem. It happens around the 50,000th read, the read times begin to gradually rise from 5ms to 200ms by the time we reach the 60,000th read. If I do a trace during this period, I see the abnormal behavior with getClosest() and next() that I described in the above comment.
So something fishy is still going on. Possibly in HStoreFile.HalfMapFileReader related to having two mapfiles.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Attachment: 2161.txt
This patch zero pads numeric rows in PE. Running a little test, the getFulls all return in a fairly constant couple of milliseconds.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541158 ]
stack commented on HADOOP-2161:
-------------------------------
Are you up trying out this patch Clint? (Again, thanks for digging in to figure what was actually going on in here).
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161-3.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Attachment: 2161-4.patch
This patch passes tests locally.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540631 ]
stack commented on HADOOP-2161:
-------------------------------
Thanks Clint. Good find. HRegion.get(...) quits after its accumulated sufficient versions -- usually one. HRegion.getFull, which is eventually called by HTable.getFull is pig-headed insisting on running through memcache and all store files for every possible version regardless of version setting.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Status: Patch Available (was: Open)
Passes tests locally.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161-3.patch, 2161-4.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2161) getRow() is orders of magnitudes
slower than get(), even on rows with one column
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2161:
--------------------------
Attachment: 2161-2.patch
memcache had same problem as hstore always scanning to the end instead of stopping once it'd fallen out of the asked-for row.
> getRow() is orders of magnitudes slower than get(), even on rows with one column
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2161
> URL: https://issues.apache.org/jira/browse/HADOOP-2161
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Environment: latest from trunk
> Reporter: Clint Morgan
> Attachments: 2161-2.patch, 2161.txt, HADOOP-2161-2.patch, PerformanceEvaluation-patch.txt
>
>
> HTable.getRow(Text) is several orders of magnitude slower than
> HTable.get(Text, Text), even on rows with a single column.
> This problem can be observed by the attached patch of
> PerformanceEvaluation.java which changes SequentialRead to use getRow,
> and prints out the time for each read.
> The test can the be run with:
> bin/hbase org.apache.hadoop.hbase.PerformaeEvaluation sequentialRead 1
> On my laptop, the original test (using get()) produces reads on the order of 5-20
> milliseconds. Using getRow(), the reads take 50-2000 ms.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.