[jira] [Created] (KYLIN-4328) When hbase and kylin that are not in the same IDC and found that the build task became very slow during scheduling.

["GuKe (Jira)" <ji...@apache.org>]

GuKe created KYLIN-4328:
---------------------------

             Summary: When hbase and kylin that are not in the same IDC and found that the build task became very slow during scheduling. 
                 Key: KYLIN-4328
                 URL: https://issues.apache.org/jira/browse/KYLIN-4328
             Project: Kylin
          Issue Type: Improvement
          Components: Job Engine
    Affects Versions: v2.6.3, v3.0.0, v2.2.0
         Environment: Centos 7.4
hbase 1.2.4
hive 1.1.1
hadoop 2.7.2
            Reporter: GuKe
            Assignee: GuKe


When hbase and kylin that are not in the same IDC and found that the build task became very slow during scheduling. 
We found that it was caused by the following part of the code.
The method getExecutableManager().GetAllJobIdsInCache() will read all of jobid,There are currently more than 35,000 jobs in our server,and each jobid accesses hbase at least twice to read the job state. 
While that the most of jobs are succeed status.Those status won't change.
When kylin and hbase services are in the same IDC each visit to hbase Network Latency is less than 1 ms.
However it takes more than 5 ms to access hbase each time across the IDC so the delay caused by accessing hbase is considerable. 
It takes a long time for scheduling task to run.
So we can add a cache to hold the id of the successful job at the first time of the service start.
After we modified the code the run time reduced from 10 minutes to 20 seconds.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)