[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

["Vikas Vishwakarma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372409#comment-16372409 ] 

Vikas Vishwakarma commented on PHOENIX-4625:
--------------------------------------------

[~larsh] [~giacomotaylor] this is one of the memory leak case observed with QueryServer 

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -----------------------------------------------------------------------------
>
>                 Key: PHOENIX-4625
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4625
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.13.0
>            Reporter: Vikas Vishwakarma
>            Priority: Major
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the following checks  if renew lease feature is supported and if the renew lease config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease renewal based on just the check if the renew lease feature is supported services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In this case PhoenixConnection will keep adding the iterators for every scan into the scannerQueue for renewal based on the feature supported check but the renewal task is not running because phoenix.scanner.lease.renew.enabled is set to false, so the scannerQueue will keep growing as long as the PhoenixConnection is alive and multiple scans requests are coming on this connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and does billions of scans on this connection. In this case scannerQueue is growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =============================================
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE)) {
>  checkNotNull(itr);
> scannerQueue.add(new WeakReference<TableResultIterator>(itr));
>  }
>  }
>  
> Starting the RenewLeaseTask
> =============================
> checks if Feature.RENEW_LEASE is supported and if phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> ConnectionQueryServicesImpl {
> ....
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_RENEW_LEASE_ENABLED);
> .....
> @Override
>  public boolean isRenewingLeasesEnabled() {
>  return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled;
>  }
> private void scheduleRenewLeaseTasks() {
> if (isRenewingLeasesEnabled()) {
> renewLeaseExecutor =
>  Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
>  for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : connectionQueues) {
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
>  renewLeaseTaskFrequency, TimeUnit.MILLISECONDS);
>  }
>  }
>  }
> .......
> }
>  
> To solve this We must add both checks in PhoenixConnection if the feature is supported and if the config is enabled before adding the iterators to scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)