You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by op <52...@qq.com> on 2018/06/08 03:08:21 UTC
ERROR WHEN Saving Cuboid Statistics
Hi，everyone,I‘am running example kylin_sales_model.
At saving cuboid statistics step
log says
details in kylin.log are below:
there isn't any other error information ,and i can't find out what's wrong,can you help me please?
my version :
kylin 2.0.0
hbase 1.2.6
hdaoop 2.9.0
hive 2.3.2 beeline
2018-06-07 16:08:18,252 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] conf.HiveConf:4116 : HiveConf of name hive.mapred.supports.subdirectories does not exist
2018-06-07 16:08:18,253 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] hive.metastore:407 : Trying to connect to metastore with URI thrift://hadoop1001:9083
2018-06-07 16:08:18,254 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] hive.metastore:481 : Opened a connection to metastore, current connections: 51
2018-06-07 16:08:18,254 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] hive.metastore:534 : Connected to metastore.
2018-06-07 16:08:18,341 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] lookup.SnapshotManager:214 : Loading snapshotTable from /table_snapshot/KYLIN.KYLIN_CAL_DT/c611bd70-b196-45ef-9edc-18d6a1126277.snapshot, with loadData: false
2018-06-07 16:08:18,384 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] lookup.SnapshotManager:130 : Identical input FileSignature [path=hdfs://beh/user/hive/warehouse/kylin.db/kylin_cal_dt, size=512997, lastModifiedTime=1528356298671], reuse existing snapshot at /table_snapshot/KYLIN.KYLIN_CAL_DT/c611bd70-b196-45ef-9edc-18d6a1126277.snapshot
2018-06-07 16:08:18,384 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cube.CubeManager:366 : Updating cube instance 'kylin_sales_cube'
2018-06-07 16:08:18,384 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cube.CubeValidator:102 : NEW segment start does not fit/connect with other segments: kylin_sales_cube[20120101000000_20121231000000]
2018-06-07 16:08:18,384 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cube.CubeValidator:104 : NEW segment end does not fit/connect with other segments: kylin_sales_cube[20120101000000_20121231000000]
2018-06-07 16:08:18,387 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:91 : Building snapshot of KYLIN.KYLIN_COUNTRY
2018-06-07 16:08:18,387 INFO [pool-11-thread-1] cachesync.Broadcaster:123 : Announcing new broadcast event: BroadcastEvent{entity=cube, event=update, cacheKey=kylin_sales_cube}
2018-06-07 16:08:18,391 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:189 : Broadcasting metadata change: entity=cube, event=UPDATE, cacheKey=kylin_sales_cube, listeners=[org.apache.kylin.cube.CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.hybrid.HybridManager$HybridSyncListener@581a3b84, org.apache.kylin.rest.service.CacheService$1@6c4e4197]
2018-06-07 16:08:18,393 INFO [http-bio-7070-exec-3] cube.CubeManager:785 : Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 1 segments
2018-06-07 16:08:18,393 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:189 : Broadcasting metadata change: entity=project_data, event=UPDATE, cacheKey=learn_kylin, listeners=[org.apache.kylin.metadata.project.ProjectManager$ProjectSyncListener@51853c91, org.apache.kylin.metadata.MetadataManager$SrcTableSyncListener@67cc135, org.apache.kylin.metadata.MetadataManager$SrcTableExtSyncListener@33eb1107, org.apache.kylin.metadata.MetadataManager$DataModelSyncListener@6d749bd4, org.apache.kylin.metadata.MetadataManager$ExtFilterSyncListener@9f57e2e, org.apache.kylin.cube.CubeDescManager$CubeDescSyncListener@71ecba35, org.apache.kylin.cube.CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.hybrid.HybridManager$HybridSyncListener@581a3b84, org.apache.kylin.metadata.realization.RealizationRegistry$1@7a3c2aa7, org.apache.kylin.rest.service.CacheService$1@6c4e4197]
2018-06-07 16:08:18,394 INFO [http-bio-7070-exec-3] service.CacheService:135 : removeOLAPDataSource is called for project learn_kylin
2018-06-07 16:08:18,394 INFO [http-bio-7070-exec-3] service.CacheService:117 : cleaning cache for project learn_kylin (currently remove all entries)
2018-06-07 16:08:18,394 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:219 : Done broadcasting metadata change: entity=project_data, event=UPDATE, cacheKey=learn_kylin
2018-06-07 16:08:18,395 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:219 : Done broadcasting metadata change: entity=cube, event=UPDATE, cacheKey=kylin_sales_cube
2018-06-07 16:08:18,511 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] conf.HiveConf:4116 : HiveConf of name hive.server2.custom.authentication.file does not exist
2018-06-07 16:08:18,512 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] conf.HiveConf:4116 : HiveConf of name hive.mapred.supports.subdirectories does not exist
2018-06-07 16:08:18,513 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] hive.metastore:407 : Trying to connect to metastore with URI thrift://hadoop1001:9083
2018-06-07 16:08:18,513 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] hive.metastore:481 : Opened a connection to metastore, current connections: 52
2018-06-07 16:08:18,514 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] hive.metastore:534 : Connected to metastore.
2018-06-07 16:08:18,595 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] lookup.SnapshotManager:214 : Loading snapshotTable from /table_snapshot/KYLIN.KYLIN_COUNTRY/02c57ee0-b8a3-47eb-bedc-ebd7d9caeb60.snapshot, with loadData: false
2018-06-07 16:08:18,639 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] lookup.SnapshotManager:130 : Identical input FileSignature [path=hdfs://beh/user/hive/warehouse/kylin.db/kylin_country, size=8537, lastModifiedTime=1528356298714], reuse existing snapshot at /table_snapshot/KYLIN.KYLIN_COUNTRY/02c57ee0-b8a3-47eb-bedc-ebd7d9caeb60.snapshot
2018-06-07 16:08:18,640 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cube.CubeManager:366 : Updating cube instance 'kylin_sales_cube'
2018-06-07 16:08:18,640 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cube.CubeValidator:102 : NEW segment start does not fit/connect with other segments: kylin_sales_cube[20120101000000_20121231000000]
2018-06-07 16:08:18,640 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cube.CubeValidator:104 : NEW segment end does not fit/connect with other segments: kylin_sales_cube[20120101000000_20121231000000]
2018-06-07 16:08:18,643 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[BUYER_COUNTRY:KYLIN_COUNTRY]
2018-06-07 16:08:18,644 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[SELLER_ACCOUNT:KYLIN_ACCOUNT]
2018-06-07 16:08:18,646 INFO [pool-11-thread-1] cachesync.Broadcaster:123 : Announcing new broadcast event: BroadcastEvent{entity=cube, event=update, cacheKey=kylin_sales_cube}
2018-06-07 16:08:18,650 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[KYLIN_CATEGORY_GROUPINGS]
2018-06-07 16:08:18,650 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[SELLER_COUNTRY:KYLIN_COUNTRY]
2018-06-07 16:08:18,651 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[BUYER_ACCOUNT:KYLIN_ACCOUNT]
2018-06-07 16:08:18,652 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:189 : Broadcasting metadata change: entity=cube, event=UPDATE, cacheKey=kylin_sales_cube, listeners=[org.apache.kylin.cube.CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.hybrid.HybridManager$HybridSyncListener@581a3b84, org.apache.kylin.rest.service.CacheService$1@6c4e4197]
2018-06-07 16:08:18,654 INFO [http-bio-7070-exec-3] cube.CubeManager:785 : Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 1 segments
2018-06-07 16:08:18,654 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:189 : Broadcasting metadata change: entity=project_data, event=UPDATE, cacheKey=learn_kylin, listeners=[org.apache.kylin.metadata.project.ProjectManager$ProjectSyncListener@51853c91, org.apache.kylin.metadata.MetadataManager$SrcTableSyncListener@67cc135, org.apache.kylin.metadata.MetadataManager$SrcTableExtSyncListener@33eb1107, org.apache.kylin.metadata.MetadataManager$DataModelSyncListener@6d749bd4, org.apache.kylin.metadata.MetadataManager$ExtFilterSyncListener@9f57e2e, org.apache.kylin.cube.CubeDescManager$CubeDescSyncListener@71ecba35, org.apache.kylin.cube.CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.hybrid.HybridManager$HybridSyncListener@581a3b84, org.apache.kylin.metadata.realization.RealizationRegistry$1@7a3c2aa7, org.apache.kylin.rest.service.CacheService$1@6c4e4197]
2018-06-07 16:08:18,654 INFO [http-bio-7070-exec-3] service.CacheService:135 : removeOLAPDataSource is called for project learn_kylin
2018-06-07 16:08:18,655 INFO [http-bio-7070-exec-3] service.CacheService:117 : cleaning cache for project learn_kylin (currently remove all entries)
2018-06-07 16:08:18,655 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:219 : Done broadcasting metadata change: entity=project_data, event=UPDATE, cacheKey=learn_kylin
2018-06-07 16:08:18,655 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:219 : Done broadcasting metadata change: entity=cube, event=UPDATE, cacheKey=kylin_sales_cube
2018-06-07 16:08:18,657 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[KYLIN_CAL_DT]
2018-06-07 16:08:18,676 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-03
2018-06-07 16:08:18,681 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-03
2018-06-07 16:08:18,683 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-03 from RUNNING to SUCCEED
2018-06-07 16:08:18,687 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
2018-06-07 16:08:18,705 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
2018-06-07 16:08:18,707 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2 from RUNNING to READY
2018-06-07 16:08:18,716 INFO [pool-8-thread-1] threadpool.DefaultScheduler:114 : CubingJob{id=6faa47ea-3a5b-4020-976f-c9fcf9d93bd2, name=kylin_sales_cube - 20120101000000_20121231000000 - BUILD - GMT+08:00 2018-06-07 16:05:02, state=READY} prepare to schedule
2018-06-07 16:08:18,716 INFO [pool-8-thread-1] threadpool.DefaultScheduler:117 : CubingJob{id=6faa47ea-3a5b-4020-976f-c9fcf9d93bd2, name=kylin_sales_cube - 20120101000000_20121231000000 - BUILD - GMT+08:00 2018-06-07 16:05:02, state=READY} scheduled
2018-06-07 16:08:18,716 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.AbstractExecutable:110 : Executing AbstractExecutable (kylin_sales_cube - 20120101000000_20121231000000 - BUILD - GMT+08:00 2018-06-07 16:05:02)
2018-06-07 16:08:18,717 INFO [pool-8-thread-1] threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 1 actual running, 0 stopped, 1 ready, 0 already succeed, 0 error, 2 discarded, 0 others
2018-06-07 16:08:18,717 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
2018-06-07 16:08:18,720 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2 from READY to RUNNING
2018-06-07 16:08:18,727 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.AbstractExecutable:110 : Executing AbstractExecutable (Save Cuboid Statistics)
2018-06-07 16:08:18,728 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-04
2018-06-07 16:08:18,730 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-04 from READY to RUNNING
2018-06-07 16:08:18,771 ERROR [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] steps.SaveStatisticsStep:82 : fail to save cuboid statistics
java.io.IOException: fail to find the statistics file in base dir: hdfs:/kylin/kylin_metadata/kylin-6faa47ea-3a5b-4020-976f-c9fcf9d93bd2/kylin_sales_cube/fact_distinct_columns/statistics
at org.apache.kylin.engine.mr.steps.SaveStatisticsStep.doWork(SaveStatisticsStep.java:66)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-06-07 16:08:18,772 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-04
2018-06-07 16:08:18,777 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-04
2018-06-07 16:08:18,779 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-04 from RUNNING to ERROR
2018-06-07 16:08:18,784 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
2018-06-07 16:08:18,788 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
2018-06-07 16:08:18,790 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
2018-06-07 16:08:18,792 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2 from RUNNING to ERROR
2018-06-07 16:08:18,792 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424] execution.AbstractExecutable:258 : no need to send email, user list is empty
2018-06-07 16:08:18,798 INFO [pool-8-thread-1] threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 2 discarded, 0 others
2018-06-07 16:08:56,561 INFO [pool-8-thread-1] threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 2 discarded, 0 others
上一封 下一封
« 返回
Re: ERROR WHEN Saving Cuboid Statistics
Posted by Li Yang <li...@apache.org>.
The missing
>
hdfs:/kylin/kylin_metadata/kylin-6faa47ea-3a5b-4020-976f-c9fcf9d93bd2/kylin_sales_cube/fact_distinct_columns/statistics
should be the output of the "Fact distinct values" step.
I would check 1) if the file is really there; 2) in kylin.log if anything
suspicious around the step of "Fact distinct values".
If find nothing, then discard and re-submit the job is a temporary
workaround.
On Fri, Jun 8, 2018 at 11:08 AM, op <52...@qq.com> wrote:
>
> Hi，everyone,I‘am running example kylin_sales_model.
>
> At saving cuboid statistics step
>
> log says
>
> details in kylin.log are below:
>
> there isn't any other error information ,and i can't find out what's
> wrong,can you help me please?
> my version :
> kylin 2.0.0
> hbase 1.2.6
> hdaoop 2.9.0
> hive 2.3.2 beeline
>
> 2018-06-07 16:08:18,252 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> conf.HiveConf:4116 : HiveConf of name hive.mapred.supports.subdirectories
> does not exist
> 2018-06-07 16:08:18,253 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> hive.metastore:407 : Trying to connect to metastore with URI
> thrift://hadoop1001:9083
> 2018-06-07 16:08:18,254 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> hive.metastore:481 : Opened a connection to metastore, current connections:
> 51
> 2018-06-07 16:08:18,254 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> hive.metastore:534 : Connected to metastore.
> 2018-06-07 16:08:18,341 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> lookup.SnapshotManager:214 : Loading snapshotTable from
> /table_snapshot/KYLIN.KYLIN_CAL_DT/c611bd70-b196-45ef-9edc-18d6a1126277.snapshot,
> with loadData: false
> 2018-06-07 16:08:18,384 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> lookup.SnapshotManager:130 : Identical input FileSignature
> [path=hdfs://beh/user/hive/warehouse/kylin.db/kylin_cal_dt, size=512997,
> lastModifiedTime=1528356298671], reuse existing snapshot at
> /table_snapshot/KYLIN.KYLIN_CAL_DT/c611bd70-b196-45ef-9edc-18d6a1126277
> .snapshot
> 2018-06-07 16:08:18,384 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cube.CubeManager:366 : Updating cube instance 'kylin_sales_cube'
> 2018-06-07 16:08:18,384 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cube.CubeValidator:102 : NEW segment start does not fit/connect with other
> segments: kylin_sales_cube[20120101000000_20121231000000]
> 2018-06-07 16:08:18,384 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cube.CubeValidator:104 : NEW segment end does not fit/connect with other
> segments: kylin_sales_cube[20120101000000_20121231000000]
> 2018-06-07 16:08:18,387 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:91 : Building snapshot of KYLIN.KYLIN_COUNTRY
> 2018-06-07 16:08:18,387 INFO [pool-11-thread-1]
> cachesync.Broadcaster:123 : Announcing new broadcast event:
> BroadcastEvent{entity=cube, event=update, cacheKey=kylin_sales_cube}
> 2018-06-07 16:08:18,391 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:189 : Broadcasting metadata change: entity=cube,
> event=UPDATE, cacheKey=kylin_sales_cube, listeners=[org.apache.kylin.
> cube.CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.
> hybrid.HybridManager$HybridSyncListener@581a3b84,
> org.apache.kylin.rest.service.CacheService$1@6c4e4197]
> 2018-06-07 16:08:18,393 INFO [http-bio-7070-exec-3] cube.CubeManager:785
> : Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 1
> segments
> 2018-06-07 16:08:18,393 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:189 : Broadcasting metadata change:
> entity=project_data, event=UPDATE, cacheKey=learn_kylin,
> listeners=[org.apache.kylin.metadata.project.ProjectManager$
> ProjectSyncListener@51853c91, org.apache.kylin.metadata.MetadataManager$
> SrcTableSyncListener@67cc135, org.apache.kylin.metadata.MetadataManager$
> SrcTableExtSyncListener@33eb1107, org.apache.kylin.metadata.
> MetadataManager$DataModelSyncListener@6d749bd4, org.apache.kylin.metadata.
> MetadataManager$ExtFilterSyncListener@9f57e2e, org.apache.kylin.cube.
> CubeDescManager$CubeDescSyncListener@71ecba35, org.apache.kylin.cube.
> CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.
> hybrid.HybridManager$HybridSyncListener@581a3b84,
> org.apache.kylin.metadata.realization.RealizationRegistry$1@7a3c2aa7,
> org.apache.kylin.rest.service.CacheService$1@6c4e4197]
> 2018-06-07 16:08:18,394 INFO [http-bio-7070-exec-3]
> service.CacheService:135 : removeOLAPDataSource is called for project
> learn_kylin
> 2018-06-07 16:08:18,394 INFO [http-bio-7070-exec-3]
> service.CacheService:117 : cleaning cache for project learn_kylin
> (currently remove all entries)
> 2018-06-07 16:08:18,394 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:219 : Done broadcasting metadata change:
> entity=project_data, event=UPDATE, cacheKey=learn_kylin
> 2018-06-07 16:08:18,395 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:219 : Done broadcasting metadata change: entity=cube,
> event=UPDATE, cacheKey=kylin_sales_cube
> 2018-06-07 16:08:18,511 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> conf.HiveConf:4116 : HiveConf of name hive.server2.custom.authentication.file
> does not exist
> 2018-06-07 16:08:18,512 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> conf.HiveConf:4116 : HiveConf of name hive.mapred.supports.subdirectories
> does not exist
> 2018-06-07 16:08:18,513 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> hive.metastore:407 : Trying to connect to metastore with URI
> thrift://hadoop1001:9083
> 2018-06-07 16:08:18,513 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> hive.metastore:481 : Opened a connection to metastore, current connections:
> 52
> 2018-06-07 16:08:18,514 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> hive.metastore:534 : Connected to metastore.
> 2018-06-07 16:08:18,595 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> lookup.SnapshotManager:214 : Loading snapshotTable from
> /table_snapshot/KYLIN.KYLIN_COUNTRY/02c57ee0-b8a3-47eb-bedc-ebd7d9caeb60.snapshot,
> with loadData: false
> 2018-06-07 16:08:18,639 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> lookup.SnapshotManager:130 : Identical input FileSignature
> [path=hdfs://beh/user/hive/warehouse/kylin.db/kylin_country, size=8537,
> lastModifiedTime=1528356298714], reuse existing snapshot at
> /table_snapshot/KYLIN.KYLIN_COUNTRY/02c57ee0-b8a3-47eb-
> bedc-ebd7d9caeb60.snapshot
> 2018-06-07 16:08:18,640 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cube.CubeManager:366 : Updating cube instance 'kylin_sales_cube'
> 2018-06-07 16:08:18,640 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cube.CubeValidator:102 : NEW segment start does not fit/connect with other
> segments: kylin_sales_cube[20120101000000_20121231000000]
> 2018-06-07 16:08:18,640 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cube.CubeValidator:104 : NEW segment end does not fit/connect with other
> segments: kylin_sales_cube[20120101000000_20121231000000]
> 2018-06-07 16:08:18,643 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:96 : Checking snapshot of
> TableRef[BUYER_COUNTRY:KYLIN_COUNTRY]
> 2018-06-07 16:08:18,644 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:96 : Checking snapshot of
> TableRef[SELLER_ACCOUNT:KYLIN_ACCOUNT]
> 2018-06-07 16:08:18,646 INFO [pool-11-thread-1]
> cachesync.Broadcaster:123 : Announcing new broadcast event:
> BroadcastEvent{entity=cube, event=update, cacheKey=kylin_sales_cube}
> 2018-06-07 16:08:18,650 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:96 : Checking snapshot of
> TableRef[KYLIN_CATEGORY_GROUPINGS]
> 2018-06-07 16:08:18,650 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:96 : Checking snapshot of
> TableRef[SELLER_COUNTRY:KYLIN_COUNTRY]
> 2018-06-07 16:08:18,651 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:96 : Checking snapshot of
> TableRef[BUYER_ACCOUNT:KYLIN_ACCOUNT]
> 2018-06-07 16:08:18,652 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:189 : Broadcasting metadata change: entity=cube,
> event=UPDATE, cacheKey=kylin_sales_cube, listeners=[org.apache.kylin.
> cube.CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.
> hybrid.HybridManager$HybridSyncListener@581a3b84,
> org.apache.kylin.rest.service.CacheService$1@6c4e4197]
> 2018-06-07 16:08:18,654 INFO [http-bio-7070-exec-3] cube.CubeManager:785
> : Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 1
> segments
> 2018-06-07 16:08:18,654 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:189 : Broadcasting metadata change:
> entity=project_data, event=UPDATE, cacheKey=learn_kylin,
> listeners=[org.apache.kylin.metadata.project.ProjectManager$
> ProjectSyncListener@51853c91, org.apache.kylin.metadata.MetadataManager$
> SrcTableSyncListener@67cc135, org.apache.kylin.metadata.MetadataManager$
> SrcTableExtSyncListener@33eb1107, org.apache.kylin.metadata.
> MetadataManager$DataModelSyncListener@6d749bd4, org.apache.kylin.metadata.
> MetadataManager$ExtFilterSyncListener@9f57e2e, org.apache.kylin.cube.
> CubeDescManager$CubeDescSyncListener@71ecba35, org.apache.kylin.cube.
> CubeManager$CubeSyncListener@11f9b854, org.apache.kylin.storage.
> hybrid.HybridManager$HybridSyncListener@581a3b84,
> org.apache.kylin.metadata.realization.RealizationRegistry$1@7a3c2aa7,
> org.apache.kylin.rest.service.CacheService$1@6c4e4197]
> 2018-06-07 16:08:18,654 INFO [http-bio-7070-exec-3]
> service.CacheService:135 : removeOLAPDataSource is called for project
> learn_kylin
> 2018-06-07 16:08:18,655 INFO [http-bio-7070-exec-3]
> service.CacheService:117 : cleaning cache for project learn_kylin
> (currently remove all entries)
> 2018-06-07 16:08:18,655 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:219 : Done broadcasting metadata change:
> entity=project_data, event=UPDATE, cacheKey=learn_kylin
> 2018-06-07 16:08:18,655 DEBUG [http-bio-7070-exec-3]
> cachesync.Broadcaster:219 : Done broadcasting metadata change: entity=cube,
> event=UPDATE, cacheKey=kylin_sales_cube
> 2018-06-07 16:08:18,657 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> cli.DictionaryGeneratorCLI:96 : Checking snapshot of TableRef[KYLIN_CAL_DT]
> 2018-06-07 16:08:18,676 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-03
> 2018-06-07 16:08:18,681 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-03
> 2018-06-07 16:08:18,683 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-03 from RUNNING to SUCCEED
> 2018-06-07 16:08:18,687 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2
> 2018-06-07 16:08:18,705 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2
> 2018-06-07 16:08:18,707 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
> from RUNNING to READY
> 2018-06-07 16:08:18,716 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:114 : CubingJob{id=6faa47ea-3a5b-4020-976f-c9fcf9d93bd2,
> name=kylin_sales_cube - 20120101000000_20121231000000 - BUILD - GMT+08:00
> 2018-06-07 16:05:02, state=READY} prepare to schedule
> 2018-06-07 16:08:18,716 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:117 : CubingJob{id=6faa47ea-3a5b-4020-976f-c9fcf9d93bd2,
> name=kylin_sales_cube - 20120101000000_20121231000000 - BUILD - GMT+08:00
> 2018-06-07 16:05:02, state=READY} scheduled
> 2018-06-07 16:08:18,716 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.AbstractExecutable:110 : Executing AbstractExecutable
> (kylin_sales_cube - 20120101000000_20121231000000 - BUILD - GMT+08:00
> 2018-06-07 16:05:02)
> 2018-06-07 16:08:18,717 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 1 actual
> running, 0 stopped, 1 ready, 0 already succeed, 0 error, 2 discarded, 0
> others
> 2018-06-07 16:08:18,717 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2
> 2018-06-07 16:08:18,720 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
> from READY to RUNNING
> 2018-06-07 16:08:18,727 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.AbstractExecutable:110 : Executing AbstractExecutable (Save
> Cuboid Statistics)
> 2018-06-07 16:08:18,728 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-04
> 2018-06-07 16:08:18,730 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-04 from READY to RUNNING
> 2018-06-07 16:08:18,771 ERROR [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> steps.SaveStatisticsStep:82 : fail to save cuboid statistics
> java.io.IOException: fail to find the statistics file in base dir:
> hdfs:/kylin/kylin_metadata/kylin-6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2/kylin_sales_cube/fact_distinct_columns/statistics
> at org.apache.kylin.engine.mr.steps.SaveStatisticsStep.
> doWork(SaveStatisticsStep.java:66)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-06-07 16:08:18,772 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-04
> 2018-06-07 16:08:18,777 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-04
> 2018-06-07 16:08:18,779 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2-04 from RUNNING to ERROR
> 2018-06-07 16:08:18,784 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2
> 2018-06-07 16:08:18,788 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2
> 2018-06-07 16:08:18,790 DEBUG [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> dao.ExecutableDao:217 : updating job output, id: 6faa47ea-3a5b-4020-976f-
> c9fcf9d93bd2
> 2018-06-07 16:08:18,792 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.ExecutableManager:389 : job id:6faa47ea-3a5b-4020-976f-c9fcf9d93bd2
> from RUNNING to ERROR
> 2018-06-07 16:08:18,792 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
> execution.AbstractExecutable:258 : no need to send email, user list is
> empty
> 2018-06-07 16:08:18,798 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 0 already succeed, 1 error, 2 discarded, 0
> others
> 2018-06-07 16:08:56,561 INFO [pool-8-thread-1]
> threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 0 already succeed, 1 error, 2 discarded, 0
> others
>
> 上一封 下一封
> « 返回
>