You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Olivier Guennec (JIRA)" <ji...@apache.org> on 2015/06/29 16:47:05 UTC
[jira] [Closed] (KYLIN-612) Job not kicking step 2 after step 1
succeeded - job status pending
[ https://issues.apache.org/jira/browse/KYLIN-612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivier Guennec closed KYLIN-612.
---------------------------------
I have managed to create a cube with the same metadata using Kylin 0.7.1 RC3 on HDP 2.2.4.2 with Hbase 0.98.4
> Job not kicking step 2 after step 1 succeeded - job status pending
> ------------------------------------------------------------------
>
> Key: KYLIN-612
> URL: https://issues.apache.org/jira/browse/KYLIN-612
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v0.6.5
> Environment: Horton Works Data platform 2.1 - 6 nodes cluster
> Similar job succeeded in v0.6.2
> Reporter: Olivier Guennec
> Assignee: ZhouQianhao
>
> Metadata from previous releases was dropped before installing Kylin v0.6.5
> Cluster libraries changed in kylin parent pom.xml
> <hadoop2.version>2.4.0.2.1.2.0-402</hadoop2.version>
> <yarn.version>2.4.0.2.1.2.0-402</yarn.version>
> <hbase-hadoop2.version>0.98.0.2.1.2.0-402-hadoop2</hbase-hadoop2.version>
> <zookeeper.version>3.4.5.2.1.2.0-402</zookeeper.version>
> <hive.version>0.13.0.2.1.2.0-402</hive.version>
> The following 2 MR jobs succeeded as part of step 1
> Job Name:
> SELECT
> FACT_SALES.UNITS_SOL...DIM_PRODUCT.ID(Stage-3)
> User Name: root
> Queue: default
> State: SUCCEEDED
> Uberized: false
> Submitted: Tue Feb 24 16:23:17 GMT 2015
> Started: Tue Feb 24 16:23:22 GMT 2015
> Finished: Tue Feb 24 16:23:28 GMT 2015
> Elapsed: 6sec
> Diagnostics:
> Average Map Time 4sec
> Job Name: INSERT OVERWRITE TABLE kyli...DIM_PRODUCT.ID(Stage-4)
> User Name: root
> Queue: default
> State: SUCCEEDED
> Uberized: false
> Submitted: Tue Feb 24 16:25:58 GMT 2015
> Started: Tue Feb 24 16:26:02 GMT 2015
> Finished: Tue Feb 24 16:26:09 GMT 2015
> Elapsed: 6sec
> Diagnostics:
> Average Map Time 4sec
> No other MR job is being launched as apparently Step 2 is not launched by Job Engine.
> The Job SQL statement when run from Hive is returning data.
> SELECT
> FACT_SALES.UNITS_SOLD
> ,FACT_SALES.PRODUCT_ID
> FROM FACT_SALES
> INNER JOIN DIM_PRODUCT
> ON FACT_SALES.PRODUCT_ID = DIM_PRODUCT.ID
> 2 92
> 1 92
> 3 91
> 9 92
> 4 93
> 11 91
> 8 91
> 7 92
> 6 91
> 3 92
> 2 93
> 10 91
> 3 92
> 9 92
> 5 91
> 2 91
> 1 92
> 2 93
> 3 92
> There is no error in Kylin.log related to a failure (except when trying to enable cube Wiki1 before 1st segment is being built).
> extract of kylin.log
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,200][DEBUG][com.kylinolap.job.JobInstanceBuilder.createBuildCubeSegmentSteps(JobInstanceBuilder.java:248)] - {
> "uuid" : "781c2f74-5e4e-4247-af01-71260f0a550e",
> "name" : "Wiki1 - FULL_BUILD - BUILD - PST 2015-02-24 08:25:08",
> "type" : "BUILD",
> "duration" : 0,
> "steps" : null,
> "submitter" : "ADMIN",
> "progress" : "NaN",
> "last_modified" : 0,
> "related_cube" : "Wiki1",
> "related_segment" : "FULL_BUILD",
> "exec_start_time" : 0,
> "exec_end_time" : 0,
> "mr_waiting" : 0,
> "job_status" : "NEW"
> }
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,200][INFO][com.kylinolap.cube.CubeManager.updateCube(CubeManager.java:257)] - Updating cube instance 'Wiki1
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,200][DEBUG][com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.java:166)] - Saving resource /cube/Wiki1.json (Store kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure)
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,207][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.java:258)] - Reloading SourceTable from folder kylin_metadata_qa(key='/table')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,271][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.java:267)] - Loaded 4 SourceTable(s)
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,271][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManager.java:243)] - Reloading SourceTable exd info from folder kylin_metadata_qa(key='/table_exd')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,316][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManager.java:253)] - Loaded 4 SourceTable EXD(s)
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,317][INFO][com.kylinolap.metadata.MetadataManager.reloadAllCubeDesc(MetadataManager.java:308)] - Reloading Cube Metadata from folder kylin_metadata_qa(key='/cube_desc')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,337][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllCubeDesc(MetadataManager.java:333)] - Loaded 1 Cube(s)
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,337][INFO][com.kylinolap.metadata.MetadataManager.reloadAllInvertedIndexDesc(MetadataManager.java:356)] - Reloading Inverted Index Desc from folder kylin_metadata_qa(key='/invertedindex_desc')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,344][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllInvertedIndexDesc(MetadataManager.java:381)] - Loaded 0 Inverted Index Desc(s)
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,369][INFO][com.kylinolap.cube.CubeManager.updateCube(CubeManager.java:262)] - Cube with 1 segments is saved
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,383][DEBUG][com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.java:166)] - Saving resource /job/781c2f74-5e4e-4247-af01-71260f0a550e (Store kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure)
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,413][DEBUG][com.kylinolap.rest.service.AclService.createAcl(AclService.java:243)] - ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: com.kylinolap.job.JobInstance; Identifier: 781c2f74-5e4e-4247-af01-71260f0a550e] created successfully.
> [http-bio-8000-exec-2]:[2015-02-24 16:25:08,435][DEBUG][com.kylinolap.rest.service.AclService.updateAcl(AclService.java:310)] - ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: com.kylinolap.job.JobInstance; Identifier: 781c2f74-5e4e-4247-af01-71260f0a550e] updated successfully.
> [Timer-0]:[2015-02-24 16:25:13,860][DEBUG][com.kylinolap.common.restclient.Broadcaster.flush(Broadcaster.java:87)] - Broadcast nodeUri: ADMIN:KYLIN@localhost:8000, type: cube, action: update, name: WIKI1
> [Timer-0]:[2015-02-24 16:25:13,861][DEBUG][com.kylinolap.common.restclient.Broadcaster.flush(Broadcaster.java:87)] - Broadcast nodeUri: ADMIN:KYLIN@localhost:8000, type: project, action: update, name: PROJECT065
> [http-bio-8000-exec-9]:[2015-02-24 16:25:13,983][DEBUG][com.kylinolap.rest.controller.CacheController.wipeCache(CacheController.java:76)] - Reload cube WIKI1 with type:cube, event type update
> [http-bio-8000-exec-9]:[2015-02-24 16:25:13,983][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.java:258)] - Reloading SourceTable from folder kylin_metadata_qa(key='/table')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-10]:[2015-02-24 16:25:13,984][DEBUG][com.kylinolap.rest.controller.CacheController.wipeCache(CacheController.java:94)] - Reload project PROJECT065 with type:project, event type update
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,047][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.java:267)] - Loaded 4 SourceTable(s)
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,048][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManager.java:243)] - Reloading SourceTable exd info from folder kylin_metadata_qa(key='/table_exd')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,093][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManager.java:253)] - Loaded 4 SourceTable EXD(s)
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,094][INFO][com.kylinolap.metadata.MetadataManager.reloadAllCubeDesc(MetadataManager.java:308)] - Reloading Cube Metadata from folder kylin_metadata_qa(key='/cube_desc')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,115][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllCubeDesc(MetadataManager.java:333)] - Loaded 1 Cube(s)
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,116][INFO][com.kylinolap.metadata.MetadataManager.reloadAllInvertedIndexDesc(MetadataManager.java:356)] - Reloading Inverted Index Desc from folder kylin_metadata_qa(key='/invertedindex_desc')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-9]:[2015-02-24 16:25:14,123][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllInvertedIndexDesc(MetadataManager.java:381)] - Loaded 0 Inverted Index Desc(s)
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,124][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.java:258)] - Reloading SourceTable from folder kylin_metadata_qa(key='/table')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,185][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.java:267)] - Loaded 4 SourceTable(s)
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,186][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManager.java:243)] - Reloading SourceTable exd info from folder kylin_metadata_qa(key='/table_exd')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,230][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManager.java:253)] - Loaded 4 SourceTable EXD(s)
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,231][INFO][com.kylinolap.metadata.MetadataManager.reloadAllCubeDesc(MetadataManager.java:308)] - Reloading Cube Metadata from folder kylin_metadata_qa(key='/cube_desc')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,250][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllCubeDesc(MetadataManager.java:333)] - Loaded 1 Cube(s)
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,251][INFO][com.kylinolap.metadata.MetadataManager.reloadAllInvertedIndexDesc(MetadataManager.java:356)] - Reloading Inverted Index Desc from folder kylin_metadata_qa(key='/invertedindex_desc')@kylin_metadata_qa@hbase:s20r500am01.dl.uk.centricaplc.com:2181:/hbase-unsecure
> [http-bio-8000-exec-10]:[2015-02-24 16:25:14,257][DEBUG][com.kylinolap.metadata.MetadataManager.reloadAllInvertedIndexDesc(MetadataManager.java:381)] - Loaded 0 Inverted Index Desc(s)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)