[jira] [Created] (NIFI-7527) AbstractKuduProcessor deadlocks after TGT refresh

["Tamas Palfy (Jira)" <ji...@apache.org>]

Tamas Palfy created NIFI-7527:
---------------------------------

             Summary: AbstractKuduProcessor deadlocks after TGT refresh 
                 Key: NIFI-7527
                 URL: https://issues.apache.org/jira/browse/NIFI-7527
             Project: Apache NiFi
          Issue Type: Bug
            Reporter: Tamas Palfy


The fix for https://issues.apache.org/jira/browse/NIFI-7453 (PutKudu kerberos issue after TGT expires) introduced a new bug: after TGT refresh the processor ends up in a deadlock.

The reason is that the onTrigger initiates a read lock:
{code:java}
    @Override
    public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
        kuduClientReadLock.lock();
        try {
            onTrigger(context, session, kuduClientR);
        } finally {
            kuduClientReadLock.unlock();
        }
    }
{code}
and while the read lock has been acquired, later (in the same stack) - if TGT refresh occurs - a write lock is attempted:
{code:java}
...
            public synchronized boolean checkTGTAndRelogin() throws LoginException {
                boolean didRelogin = super.checkTGTAndRelogin();

                if (didRelogin) {
                    createKuduClient(context);
                }

                return didRelogin;
            }
...

    protected void createKuduClient(ProcessContext context) {
        kuduClientWriteLock.lock();
        try {
            if (this.kuduClientR.get() != null) {
                try {
                    this.kuduClientR.get().close();
                } catch (KuduException e) {
                    getLogger().error("Couldn't close Kudu client.");
                }
            }

            if (kerberosUser != null) {
                final KerberosAction<KuduClient> kerberosAction = new KerberosAction<>(kerberosUser, () -> buildClient(context), getLogger());
                this.kuduClientR.set(kerberosAction.execute());
            } else {
                this.kuduClientR.set(buildClient(context));
            }
        } finally {
            kuduClientWriteLock.unlock();
        }
    }
{code}
This write lock will stuck waiting to the previous read lock to get release.
(Other threads may have acquired the same readlock but they can release it eventually - unless they too try to acquire the write lock themselves.)

The fix should be fairly simple: need to release the read lock before trying to acquire the write lock.
Need to take care of the following though:
* Need to reacquire the read lock after the new Kudu client has been created because the current logic calls an unlock on the ReadLock object later in a finally block. As this is a reentrant lock, that means decreasing the counter for the number of locks. We need to make sure the that number of locks and unlocks match.
* Need to make sure whenever we reference the Kudu client, it's actually the latest one. volatile no longer suffice, we need to wrap it in an AtomicReference.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)