RE: ATS Hook for Hive Query Text

[Xiaoyong Zhu <xi...@microsoft.com>]

+Hive users in case there are some experts there...

Xiaoyong

From: Joe Zhang (SDE)
Sent: Friday, April 10, 2015 4:57 PM
To: user@tez.apache.org
Cc: Xiaoyong Zhu
Subject: ATS Hook for Hive Query Text

Hi Hive&Tez experts:

I was using ws/v1/timeline/HIVE_QUERY_ID/ AST APIs to get tez query text for each DAG, but I am not sure whether the code has been optimized or it is same with the previous user-submmited hive code?

I has read hive<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08>/ql<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql>/src<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src>/java<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java>/org<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java/org>/apache<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java/org/apache>/hadoop<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java/org/apache/hadoop>/hive<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java/org/apache/hadoop/hive>/ql<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java/org/apache/hadoop/hive/ql>/hooks<https://github.com/apache/hive/tree/8c4c158d40b0ac3b17fe1e7f01e3e9d6fb02fe08/ql/src/java/org/apache/hadoop/hive/ql/hooks>/ATSHook.java source code and confuse whether the QueryPlan has been optimized  QueryPlan plan = hookContext.getQueryPlan();

@Override


  public void run(final HookContext hookContext) throws Exception {


    final long currentTime = System.currentTimeMillis();


    final HiveConf conf = new HiveConf(hookContext.getConf());





    executor.submit(new Runnable() {


        @Override


        public void run() {


          try {


            QueryPlan plan = hookContext.getQueryPlan();


            if (plan == null) {


              return;


            }


            String queryId = plan.getQueryId();


            String opId = hookContext.getOperationId();


            long queryStartTime = plan.getQueryStartTime();


            String user = hookContext.getUgi().getUserName();


            String requestuser = hookContext.getUserName();


            if (hookContext.getUserName() == null ){


              requestuser = hookContext.getUgi().getUserName() ;


            }


            int numMrJobs = Utilities.getMRTasks(plan.getRootTasks()).size();


            int numTezJobs = Utilities.getTezTasks(plan.getRootTasks()).size();


            if (numMrJobs + numTezJobs <= 0) {


              return; // ignore client only queries


            }





            switch(hookContext.getHookType()) {


            case PRE_EXEC_HOOK:


              ExplainTask explain = new ExplainTask();


              explain.initialize(conf, plan, null);


              String query = plan.getQueryStr();


              List<Task<?>> rootTasks = plan.getRootTasks();


              JSONObject explainPlan = explain.getJSONPlan(null, null, rootTasks,


                   plan.getFetchTask(), true, false, false);


              fireAndForget(conf, createPreHookEvent(queryId, query,


                   explainPlan, queryStartTime, user, requestuser, numMrJobs, numTezJobs, opId));


              break;


            case POST_EXEC_HOOK:


              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, requestuser, true, opId));


              break;


            case ON_FAILURE_HOOK:


              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, requestuser , false, opId));


              break;


            default:


              //ignore


              break;


            }


          } catch (Exception e) {


            LOG.info("Failed to submit plan to ATS: " + StringUtils.stringifyException(e));


          }


        }


      });


  }


Best wishes
Joe zhang