You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@linkis.apache.org by GitBox <gi...@apache.org> on 2022/04/12 02:35:53 UTC
[GitHub] [incubator-linkis] legendtkl opened a new pull request, #1949: add undone job query cache and add api /listundone
legendtkl opened a new pull request, #1949:
URL: https://github.com/apache/incubator-linkis/pull/1949
### What is the purpose of the change
refer issue: https://github.com/apache/incubator-linkis/issues/1935
### Brief change log
- Add daily scheduled refresh undone job at 00:15:00 to get the minimum id of undone job
- add api /listundone to get the undone job list, and this api will use the minimum id of undone job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on a diff in pull request #1949: add undone job query cache and add api /listundone
Posted by GitBox <gi...@apache.org>.
peacewong commented on code in PR #1949:
URL: https://github.com/apache/incubator-linkis/pull/1949#discussion_r848178417
##########
linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/cache/impl/DefaultQueryCacheManager.java:
##########
@@ -42,9 +48,13 @@ public class DefaultQueryCacheManager implements QueryCacheManager, Initializing
@Autowired private SchedulerFactoryBean schedulerFactoryBean;
+ @Autowired private JobHistoryMapper jobHistoryMapper;
+
private Map<String, Cache<String, UserTaskResultCache>> engineUserCaches =
Maps.newConcurrentMap();
+ private Long undoneTaskMinId = 0L;
Review Comment:
It is recommended to optimize the configurable hereļ¼default 0
##########
linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/cache/impl/DefaultQueryCacheManager.java:
##########
@@ -119,6 +145,30 @@ public void refreshAll() {
foreach(UserTaskResultCache::refresh);
}
+ @Override
+ public void refreshUndoneTask() {
+ PageHelper.startPage(1, 10);
+ List<JobHistory> queryTasks = null;
+ try {
+ queryTasks =
+ jobHistoryMapper.searchWithIdOrderAsc(
+ null,
Review Comment:
It is recommended to pass in undoneTaskMinId here to speed up the query
##########
linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java:
##########
@@ -190,4 +194,75 @@ public Message list(
.data(TaskConstant.TASKS, vos)
.data(JobRequestConstants.TOTAL_PAGE(), total);
}
+
+ /** Method list should not contain subjob, which may cause performance problems. */
+ @RequestMapping(path = "/listundone", method = RequestMethod.GET)
+ public Message listundone(
+ HttpServletRequest req,
+ @RequestParam(value = "startDate", required = false) Long startDate,
+ @RequestParam(value = "endDate", required = false) Long endDate,
+ @RequestParam(value = "status", required = false) String status,
+ @RequestParam(value = "pageNow", required = false) Integer pageNow,
+ @RequestParam(value = "pageSize", required = false) Integer pageSize,
+ @RequestParam(value = "startTaskID", required = false) Long taskID,
+ @RequestParam(value = "engineType", required = false) String engineType,
+ @RequestParam(value = "creator", required = false) String creator)
+ throws IOException, QueryException {
+ String username = SecurityFilter.getLoginUsername(req);
+ if (StringUtils.isEmpty(status)) {
+ status = null;
Review Comment:
The status here should be undone by default
Arrays.asList("Running", "Inited", "Scheduled")
##########
linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/cache/impl/DefaultQueryCacheManager.java:
##########
@@ -119,6 +145,30 @@ public void refreshAll() {
foreach(UserTaskResultCache::refresh);
}
+ @Override
+ public void refreshUndoneTask() {
+ PageHelper.startPage(1, 10);
+ List<JobHistory> queryTasks = null;
+ try {
+ queryTasks =
+ jobHistoryMapper.searchWithIdOrderAsc(
+ null,
+ null,
+ Arrays.asList("Running", "Inited", "Scheduled"),
+ null,
+ null,
+ null);
+ } finally {
+ PageHelper.clearPage();
+ }
+
+ PageInfo<JobHistory> pageInfo = new PageInfo<>(queryTasks);
+ List<JobHistory> list = pageInfo.getList();
+ if (!list.isEmpty()) {
+ undoneTaskMinId = list.get(0).getId();
Review Comment:
It is recommended to print the log here and print the modified value
##########
linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/cache/impl/DefaultQueryCacheManager.java:
##########
@@ -84,6 +94,22 @@ public void afterPropertiesSet() throws Exception {
logger.info("Submitted cache 00:00 refresh job.");
}
+ CronScheduleBuilder refreshUndoneJobBuilder =
+ CronScheduleBuilder.dailyAtHourAndMinute(0, 15);
Review Comment:
It is recommended to optimize the configurable hereļ¼default 0:15
##########
linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/dao/impl/JobHistoryMapper.xml:
##########
@@ -88,10 +88,23 @@
<if test="engineType != null">and engine_type = #{engineType}</if>
<if test="startDate != null">and created_time >= #{startDate} AND created_time <![CDATA[<=]]> #{endDate}</if>
<if test="status != null">and <foreach collection="status" item="element" close=")" separator="," open="status in (">#{element}</foreach></if>
+ <if test="startId != null">and id >= #{startId}</if>
</where>
ORDER BY linkis_ps_job_history_group_history.created_time DESC
</select>
+ <select id="searchWithIdOrderAsc" useCache="true" resultMap="jobHistoryMap" >
+ /*slave*/ SELECT * FROM linkis_ps_job_history_group_history
+ <where>
+ <if test="id != null">id = #{id}</if>
Review Comment:
It is recommended to change this to >=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong merged pull request #1949: add undone job query cache and add api /listundone
Posted by GitBox <gi...@apache.org>.
peacewong merged PR #1949:
URL: https://github.com/apache/incubator-linkis/pull/1949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org