[jira] [Created] (IMPALA-6640) profile collection via standard Impyla usage won't have ExecSummary

["Michael Brown (JIRA)" <ji...@apache.org>]

Michael Brown created IMPALA-6640:
-------------------------------------

             Summary: profile collection via standard Impyla usage won't have ExecSummary
                 Key: IMPALA-6640
                 URL: https://issues.apache.org/jira/browse/IMPALA-6640
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 2.11.0
            Reporter: Michael Brown


It seems the Impyla (HS2 client for Python stress test) DB API paradigm doesn't lend itself to the ideal collection of query profiles.
 # It seems a profile will not have an ExecSummary, End Time, or Errors unless the profile is requested after a CloseOperation RPC. This makes sense to me.
 # GetRuntimeProfile RPC must be called before the CloseSession RPC. This makes sense to me.
 # One can't really use standard, public methods in Impyla to do this: impala.hiveserver2.HiveServer2Cursor().close() handles both the CloseOperation and CloseSession RPCs in a single call.
 # The same is true of the cursor in a context manager \_\_exit\_\_ setting. \_\_exit\_\_ just calls close()

This means we have to tap into the more private bits of Impyla to do this.
{noformat}
#!/usr/bin/env impala-python

import impala.dbapi
from impala._thrift_api import TGetRuntimeProfileReq

with impala.dbapi.connect(host='localhost', port=21050) as conn:
  with conn.cursor() as cursor:
    cursor.execute('SELECT COUNT(2) FROM functional.alltypes')
    cursor.fetchall()
    with open('profile-before-close-operation.txt', 'w') as fh:
      fh.write(cursor.get_profile())
    # Normal cursor context would end here, but....
    # Issue CloseOperation RPC without CloseSession RPC.
    # This is impala.hiveserver2.Operation().close()
    resp = cursor._last_operation.close()
    req = TGetRuntimeProfileReq(operationHandle=cursor._last_operation.handle,
                                sessionHandle=cursor._last_operation.session.handle)
    resp = cursor._last_operation._rpc('GetRuntimeProfile', req)
    # impala.hiveserver2.HiveServer2Cursor()._reset_state() assist
    cursor._last_operation_active = False
    with open('profile-after-close-operation.txt', 'w') as fh:
      fh.write(resp.profile)
{noformat}

We should think about pythonic ways to make this better for Impyla, or at least alter the stress test to get the profiles of the queries after CloseOperation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)