You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2012/09/19 19:54:08 UTC
[jira] [Created] (HBASE-6841) Meta prefetching is slower than doing
multiple meta lookups
Jean-Daniel Cryans created HBASE-6841:
-----------------------------------------
Summary: Meta prefetching is slower than doing multiple meta lookups
Key: HBASE-6841
URL: https://issues.apache.org/jira/browse/HBASE-6841
Project: HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Priority: Critical
Fix For: 0.94.2
I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459893#comment-13459893 ]
Andrew Purtell commented on HBASE-6841:
---------------------------------------
Given J-D's observations, perhaps we should default meta prefetch to off, like Stack suggests.
Some time ago I patched our private Frankenbase to allow disable of meta prefetch on a per table basis. This was because meta prefetch was causing heap limited MR clients to OOME, and for that particular application table prefetch wasn't helpful.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6841) Meta prefetching is slower than doing
multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6841:
---------------------------------
Priority: Major (was: Critical)
Fix Version/s: (was: 0.94.3)
0.94.4
Moving to 0.94.4
[~jdcryans] Have you seen this again?
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Fix For: 0.94.4
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460254#comment-13460254 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
Haven't been able to track down that test failure, yet. It shouldn't happen, but yet somehow it does.
@J-D: Since this is (presumably) a long standing condition, how do you feel about moving this to 0.94.3?
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459757#comment-13459757 ]
Jean-Daniel Cryans commented on HBASE-6841:
-------------------------------------------
bq. ensureZookeeperTrackers only does any work when ZK was not setup already.
Right, but since we always close the connection at the end of execute() we always end up re-creating it.
bq. Did you disable this path and found faster?
No, currently it's just an educated observation that's things are broken.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6841) Meta prefetching is slower than doing
multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6841:
---------------------------------
Attachment: 6841-0.96.txt
0.96 patch for Hadoop QA
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460024#comment-13460024 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
Heh. TestHCM.testRegionCaching looks relevant :) Looking.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6841) Meta prefetching is slower than doing
multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6841:
---------------------------------
Attachment: 6841-0.94.txt
Trivial patch.
For 0.94. Looks like trunk has the same problem.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6841) Meta prefetching is slower than doing
multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6841:
---------------------------------
Assignee: Lars Hofhansl
Status: Patch Available (was: Open)
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460023#comment-13460023 ]
stack commented on HBASE-6841:
------------------------------
+1 on patch but whats that TestHCM fail about?
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458898#comment-13458898 ]
Jean-Daniel Cryans commented on HBASE-6841:
-------------------------------------------
I saw while jstack'ing that we're spending a lot of time in ensureZookeeperTrackers:
{noformat}
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
- locked <0x000000057cad4918> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1016)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
- locked <0x000000057cad4808> (a org.apache.hadoop.hbase.MasterAddressTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
- locked <0x000000057cac1258> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:827)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160)
at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:903)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:958)
- locked <0x00000005848519f0> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1492)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1377)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:702)
{noformat}
Then it would do other stuff like:
{noformat}
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
- locked <0x0000000580dd7158> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1016)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
- locked <0x0000000580dd7048> (a org.apache.hadoop.hbase.MasterAddressTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
...
{noformat}
Then we finish creating the HTable:
{noformat}
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:467)
at org.apache.hadoop.hbase.zookeeper.ClusterId.readClusterIdZNode(ClusterId.java:62)
at org.apache.hadoop.hbase.zookeeper.ClusterId.getId(ClusterId.java:50)
at org.apache.hadoop.hbase.zookeeper.ClusterId.hasId(ClusterId.java:44)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1287)
- locked <0x000000057d819bf8> (a java.lang.String)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1250)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1237)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:946)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
...
{noformat}
Then the actual pre-fetching:
{noformat}
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1007)
- locked <0x000000057ed6cf90> (a org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy11.openScanner(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:218)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:120)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:206)
{noformat}
Then we close the connection:
{noformat}
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
- locked <0x000000057f29bdc8> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1229)
at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:672)
- locked <0x000000057e9291d8> (a org.apache.zookeeper.ZooKeeper)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.close(RecoverableZooKeeper.java:553)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.close(ZooKeeperWatcher.java:452)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1751)
at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:273)
- locked <0x0000000583b59498> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:257)
- locked <0x0000000583b59498> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
at org.apache.hadoop.hbase.client.HConnectionManager.access$300(HConnectionManager.java:133)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1759)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:365)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:903)
...
{noformat}
Then do it all over again:
{noformat}
- locked <0x000000057f16eab8> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:284)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:83)
- locked <0x000000057f16dae0> (a org.apache.hadoop.hbase.MasterAddressTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
- locked <0x000000057f149dc8> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:827)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160)
at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:903)
{noformat}
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459962#comment-13459962 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
Yeah the stuff we in HTable is a disaster (if you ask me... probably has to do with multiple threads using HTables that share the same HConnection, not sure), but I think that's for another patch.
The meaning of the flag is not changed, just the default (unless I made mistake in the patch).
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460021#comment-13460021 ]
Hadoop QA commented on HBASE-6841:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545967/6841-0.96.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 139 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2909//console
This message is automatically generated.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6841) Meta prefetching is slower than doing
multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6841:
---------------------------------
Fix Version/s: (was: 0.94.2)
0.94.3
I'd like to move this to 0.94.3. Please pull back if you disagree.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.3
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460619#comment-13460619 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
Which client did you jstack? There're at least two clients at work here: The one the attempts the truncate and the ones that still perform the requests.
I assume the stuck ones are the those that are still performing the requests.
I am beginning to doubt that prefetching is a issue here. This has to do with how these clients connect. There should not need to be a connection setup each time.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459952#comment-13459952 ]
stack commented on HBASE-6841:
------------------------------
Its kinda ugly we have setRegionCachePrefetch up in HTable... Is the meaning of the enable flag up in this public API changed by this patch? Patch looks good otherwise.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460069#comment-13460069 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
Not sure what the test issue is, yet.
Also looking at the code again I notice the prefetchRegionLimit is already defaulted to 10.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484210#comment-13484210 ]
Jean-Daniel Cryans commented on HBASE-6841:
-------------------------------------------
I'm not seeing it again because I disabled prefetching in our production clusters.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Fix For: 0.94.4
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458897#comment-13458897 ]
stack commented on HBASE-6841:
------------------------------
Turn it off by default. Enable it again when we have hbase-2600 committed when it makes sense.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459753#comment-13459753 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
ensureZookeeperTrackers only does any work when ZK was not setup already. Previously this was sprinkled all over, but did the same amount of work. That part is OK, I think (in 0.94 HConnection needs a working ZK connection to function).
Just checked, HConnectionManager.execute is called all over the place (12 different places in HConnectionManager and HTable).
We do some weird stuff. For example setting the prefetching is also done through execute (i.e. get a - potentially - new HConntionImplementation, call setPrefetching on it, then close it).
On the face of it, this does not look like it's specific to region prefetching, but the fact that repeatedly do something with a new connection.
@J-D: Did you disable this path and found faster?
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459910#comment-13459910 ]
Lars Hofhansl commented on HBASE-6841:
--------------------------------------
I'd be OK with that. I'd also be worried that this is just a symptom. HConnectionManager.getConnection(...) and HConnection.close() should just do some reference counting rather than actually creating/destroying connection; which means sometime we're coming in there with a new Configuration every time...? And even that should be handled by the Configuration equivalence code we're using now.
So if, in this case, we'd remove prefetching, we'd still have the expensive Connection setup every time.
Then again and just to state the obvious, the prefetching is only useful for long lived connections and then only if these connections actually use a larg'ish portion of the prefetched entries (otherwise we're doing a lot of unnecessary work cache, and wasting memory).
Let's just disable it by default. I guess we'd do that by reversing the meaning (and name) of regionCachePrefetchDisabledTables). Happy to make a patch if you folks agree.
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6841) Meta prefetching is slower than
doing multiple meta lookups
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460674#comment-13460674 ]
Jean-Daniel Cryans commented on HBASE-6841:
-------------------------------------------
bq. @J-D: Since this is (presumably) a long standing condition, how do you feel about moving this to 0.94.3?
I'm ok.
bq. Which client did you jstack? There're at least two clients at work here: The one the attempts the truncate and the ones that still perform the requests.
The clients. I truncated separately.
bq. I am beginning to doubt that prefetching is a issue here. This has to do with how these clients connect. There should not need to be a connection setup each time.
Something is broken, we agree there :)
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.94.3
>
> Attachments: 6841-0.94.txt, 6841-0.96.txt
>
>
> I got myself into a situation where I needed to truncate a massive table while it was getting hits and surprisingly the clients were not recovering. What I see in the logs is that every time we prefetch .META. we setup a new HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira