You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "kangkaisen (JIRA)" <ji...@apache.org> on 2017/04/14 08:53:41 UTC
[jira] [Commented] (KYLIN-2547) Fix the bug of multi-process
concurrence in mergeCubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968799#comment-15968799 ]
kangkaisen commented on KYLIN-2547:
-----------------------------------
The issue reason is keepCubeRetention() and mergeCubeSegment() *only supports multithreading concurrence* and doesn't support multi-process concurrence.
So, we should only allow the job server keeping the segment lock to handle the retention check and segment auto merge.
> Fix the bug of multi-process concurrence in mergeCubeSegment
> ------------------------------------------------------------
>
> Key: KYLIN-2547
> URL: https://issues.apache.org/jira/browse/KYLIN-2547
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
> Affects Versions: v2.0.0
> Reporter: kangkaisen
> Assignee: kangkaisen
> Priority: Minor
> Fix For: v2.0.0
>
>
> There is a minor bug in "Update Cube Info" when build a cube and in distributed env.
> {code:java}
> Caused by: java.lang.IllegalStateException: Segments overlap: waimai_dolphin_topic_flow_activity_expose_food_d_cube[20170405000000_20170412000000] and waimai_dolphin_topic_flow_activity_expose_food_d_cube[20170405000000_20170412000000]
> at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85)
> at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359)
> at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386)
> at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302)
> at org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533)
> at org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635)
> at org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587)
> at org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629)
> at org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady()
> at org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237)
> at org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62)
> at org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86)
> at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305)
> at org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735)
> at org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)