You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by 김연섭 <ni...@gmail.com> on 2016/12/29 03:36:59 UTC

Zookeeper connection remains alive after DefaultStatisticsCollector's collecting stats done

In every minor compaction job of HBase,

org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidePostDepth()
is called,

and SYSTEM.CATALOG table is open to get guidepost width via

htable = env.getTable(

                    SchemaUtil.*getPhysicalTableName*
(PhoenixDatabaseMetaData.*SYSTEM_CATALOG_NAME_BYTES*, env
.getConfiguration()));

This function call creates one zookeeper connection to get cluster id.

DefaultStatisticsCollector doesn't close this zookeeper connection
immediately after get guidepost width, and the zookeeper connection remains
alive until HRegion is closed.

This is not a problem with small number of Regions, but when number of
Region is large and upsert operation is frequent, the number of zookeeper
connection gradually increases  to hundreds, and the zookeeper server nodes
experience  short of available TCP/IP ports.

I think this zookeeper connection should be closed immediately after get
guidepost width.

My Phoenix environment is,

- HBase : v1.1.8

- Phoenix : v4.9.0-HBase-1.1

- 80 nodes of hbase

The following is the Dump stack of zookeeper connection creation


       at java.lang.Thread.dumpStack(Thread.java:1329)

        at
org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:48)

        at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1663)

        at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104)

        at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:885)

        at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:638)

        at
org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:99)

        at
org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:89)

        at
org.apache.hadoop.hbase.client.CoprocessorHConnection.getConnectionForEnvironment(CoprocessorHConnection.java:61)

        at
org.apache.hadoop.hbase.client.HTableWrapper.createWrapper(HTableWrapper.java:69)

        at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:514)

        at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:503)

        at
org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidepostDepth(DefaultStatisticsCollector.java:123)

        at
org.apache.phoenix.schema.stats.DefaultStatisticsCollector.init(DefaultStatisticsCollector.java:299)

        at
org.apache.phoenix.schema.stats.DefaultStatisticsCollector.getInternalScanner(DefaultStatisticsCollector.java:293)

        at
org.apache.phoenix.schema.stats.DefaultStatisticsCollector.createCompactionScanner(DefaultStatisticsCollector.java:285)

        at
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:692)

        at
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:681)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at
org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)

        at
org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)

        at
org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:205)

        at
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.preCompact(UngroupedAggregateRegionObserver.java:681)

        at
org.apache.hadoop.hbase.coprocessor.BaseRegionObserver.preCompact(BaseRegionObserver.java:195)

        at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$9.call(RegionCoprocessorHost.java:612)

        at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1708)

        at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1785)

        at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1757)

        at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCompact(RegionCoprocessorHost.java:607)

        at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.postCreateCoprocScanner(Compactor.java:247)

        at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:91)

        at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:119)

        at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1221)

        at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1845)

        at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:521)

        at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:557)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Re: Zookeeper connection remains alive after DefaultStatisticsCollector's collecting stats done

Posted by James Taylor <ja...@apache.org>.
This definitely sounds like a serious problem. Agreed, it should be closed
immediately. Also, a minor compaction shouldn't trigger any stats
collection logic. Please file a JIRA and we'll get this fixed in the next
release. As always, patches are welcome.
Thanks,
James

On Wed, Dec 28, 2016 at 7:37 PM 김연섭 <ni...@gmail.com> wrote:

> In every minor compaction job of HBase,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidePostDepth()
> is called,
>
> and SYSTEM.CATALOG table is open to get guidepost width via
>
> htable = env.getTable(
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>                     SchemaUtil.*getPhysicalTableName*
> (PhoenixDatabaseMetaData.*SYSTEM_CATALOG_NAME_BYTES*, env
> .getConfiguration()));
>
> This function call creates one zookeeper connection to get cluster id.
>
> DefaultStatisticsCollector doesn't close this zookeeper connection
> immediately after get guidepost width, and the zookeeper connection remains
> alive until HRegion is closed.
>
> This is not a problem with small number of Regions, but when number of
> Region is large and upsert operation is frequent, the number of zookeeper
> connection gradually increases  to hundreds, and the zookeeper server nodes
> experience  short of available TCP/IP ports.
>
> I think this zookeeper connection should be closed immediately after get
> guidepost width.
>
> My Phoenix environment is,
>
> - HBase : v1.1.8
>
> - Phoenix : v4.9.0-HBase-1.1
>
> - 80 nodes of hbase
>
> The following is the Dump stack of zookeeper connection creation
>
>
>        at java.lang.Thread.dumpStack(Thread.java:1329)
>
>         at
> org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:48)
>
>         at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1663)
>
>         at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104)
>
>         at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:885)
>
>         at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:638)
>
>         at
> org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:99)
>
>         at
> org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:89)
>
>         at
> org.apache.hadoop.hbase.client.CoprocessorHConnection.getConnectionForEnvironment(CoprocessorHConnection.java:61)
>
>         at
> org.apache.hadoop.hbase.client.HTableWrapper.createWrapper(HTableWrapper.java:69)
>
>         at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:514)
>
>         at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:503)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidepostDepth(DefaultStatisticsCollector.java:123)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.init(DefaultStatisticsCollector.java:299)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.getInternalScanner(DefaultStatisticsCollector.java:293)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.createCompactionScanner(DefaultStatisticsCollector.java:285)
>
>         at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:692)
>
>         at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:681)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
>         at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
>
>         at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:498)
>
>         at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
>
>         at
> org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:205)
>
>         at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.preCompact(UngroupedAggregateRegionObserver.java:681)
>
>         at
> org.apache.hadoop.hbase.coprocessor.BaseRegionObserver.preCompact(BaseRegionObserver.java:195)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$9.call(RegionCoprocessorHost.java:612)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1708)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1785)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1757)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCompact(RegionCoprocessorHost.java:607)
>
>         at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.postCreateCoprocScanner(Compactor.java:247)
>
>         at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:91)
>
>         at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:119)
>
>         at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1221)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1845)
>
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:521)
>
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:557)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>