You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 张绚 <zh...@czkj1010.com> on 2017/07/03 09:01:45 UTC
StorageCleanup error
Hi
My test hadoop storage is almost full, so I use StorageCleanupJob to clear the unused data, but got the following error, please help analyze it and give me some suggestions, My kylin has upgraded from 1.6 to 2.0 now. Thanks.
2017-07-03 16:33:36,111 DEBUG [main MetadataManager:384]: Loaded 36 SourceTable EXD(s)
2017-07-03 16:33:36,111 DEBUG [main MetadataManager:529]: Reloading DataModel from folder kylin_metadata(key='/model_desc')@kylin_metadata@hbase
2017-07-03 16:33:36,184 ERROR [main MetadataManager:538]: Error to load DataModel at /model_desc/test_model.json
java.lang.IllegalStateException: Error to load /model_desc/test_model.json
at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:558)
at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.java:536)
at org.apache.kylin.metadata.MetadataManager.init(MetadataManager.java:284)
at org.apache.kylin.metadata.MetadataManager.<init>(MetadataManager.java:113)
at org.apache.kylin.metadata.MetadataManager.getInstance(MetadataManager.java:83)
at org.apache.kylin.job.dao.ExecutableDao.<init>(ExecutableDao.java:76)
at org.apache.kylin.job.dao.ExecutableDao.getInstance(ExecutableDao.java:57)
at org.apache.kylin.job.execution.ExecutableManager.<init>(ExecutableManager.java:55)
at org.apache.kylin.job.execution.ExecutableManager.getInstance(ExecutableManager.java:64)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.<clinit>(StorageCleanupJob.java:78)
Caused by: java.lang.IllegalArgumentException: Column not found by KYLIN_SALES.USER_ID
at org.apache.kylin.metadata.model.DataModelDesc.findColumn(DataModelDesc.java:240)
at org.apache.kylin.metadata.model.ModelDimensionDesc.init(ModelDimensionDesc.java:66)
at org.apache.kylin.metadata.model.DataModelDesc.initDimensionsAndMetrics(DataModelDesc.java:381)
at org.apache.kylin.metadata.model.DataModelDesc.init(DataModelDesc.java:302)
at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:554)
... 9 more
2017-07-03 16:33:36,194 DEBUG [main MetadataManager:543]: Loaded 8 DataModel(s)
……
2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table OK
2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table Time taken: 1.566 seconds
2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table OK
2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_dim_order_one_6b1de133_a10b_4bb6_86a7_f249e6d0c418
Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.util.StorageCleanupJob
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:362)
Caused by: java.lang.NullPointerException
at org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:84)
at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76)
at org.apache.kylin.job.execution.ExecutableManager.parseTo(ExecutableManager.java:439)
at org.apache.kylin.job.execution.ExecutableManager.getJob(ExecutableManager.java:129)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(StorageCleanupJob.java:347)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(StorageCleanupJob.java:153)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 1 more
2017-07-03 16:33:43,342 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2259]: Closing master protocol: MasterService
2017-07-03 16:33:43,368 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1830]: Closing zookeeper sessionid=0x25cbe7e555d9dc5
2017-07-03 16:33:43,371 INFO [Thread-0 ZooKeeper:684]: Session: 0x25cbe7e555d9dc5 closed
2017-07-03 16:33:43,371 INFO [main-EventThread ClientCnxn:512]: EventThread shut down
Best regards.
Re: StorageCleanup error
Posted by Billy Liu <bi...@apache.org>.
What's the new log? Did you call kylin.sh
org.apache.kylin.tool.StorageCleanupJob
--delete true?
2017-07-04 15:48 GMT+08:00 ZhangXuan <zh...@czkj1010.com>:
> I build new version and test it ,but the error still exist, so I use 1.6
> to clean up first, and next run in 2.0.0, error disappeared.
>
> Maybe the NullPointerException bug did not fixed in new version.
>
> Thanks
>
> 在 2017年7月4日,09:20,ShaoFeng Shi <sh...@apache.org> 写道:
>
> It is easy, you can follow the doc to build a new package:
> https://kylin.apache.org/development/howto_package.html
>
>
> 2017-07-03 22:53 GMT+08:00 张绚 <zh...@czkj1010.com>:
>
>> Hi,
>> I has read it ,but 2.0.0 is the last release version on website,
>> someone can give me new version package or jar files. Thanks.
>>
>> 在 2017年7月3日,22:32,ShaoFeng Shi <sh...@apache.org> 写道:
>>
>> Hi xuan,
>>
>> Check this:
>>
>> http://apache-kylin.74782.x6.nabble.com/errors-when-run-Stor
>> ageCleanupJob-td8321.html
>>
>> 2017-07-03 17:01 GMT+08:00 张绚 <zh...@czkj1010.com>:
>>
>>> Hi
>>> My test hadoop storage is almost full, so I use StorageCleanupJob to
>>> clear the unused data, but got the following error, please help analyze it
>>> and give me some suggestions, My kylin has upgraded from 1.6 to 2.0 now.
>>> Thanks.
>>>
>>> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:384]: Loaded 36
>>> SourceTable EXD(s)
>>> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:529]: Reloading
>>> DataModel from folder kylin_metadata(key='/model_des
>>> c')@kylin_metadata@hbase
>>> 2017-07-03 16:33:36,184 ERROR [main MetadataManager:538]: Error to load
>>> DataModel at /model_desc/test_model.json
>>> java.lang.IllegalStateException: Error to load
>>> /model_desc/test_model.json
>>> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDes
>>> cAt(MetadataManager.java:558)
>>> at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel
>>> (MetadataManager.java:536)
>>> at org.apache.kylin.metadata.MetadataManager.init(MetadataManag
>>> er.java:284)
>>> at org.apache.kylin.metadata.MetadataManager.<init>(MetadataMan
>>> ager.java:113)
>>> at org.apache.kylin.metadata.MetadataManager.getInstance(Metada
>>> taManager.java:83)
>>> at org.apache.kylin.job.dao.ExecutableDao.<init>(ExecutableDao.
>>> java:76)
>>> at org.apache.kylin.job.dao.ExecutableDao.getInstance(Executabl
>>> eDao.java:57)
>>> at org.apache.kylin.job.execution.ExecutableManager.<init>(Exec
>>> utableManager.java:55)
>>> at org.apache.kylin.job.execution.ExecutableManager.getInstance
>>> (ExecutableManager.java:64)
>>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.<clini
>>> t>(StorageCleanupJob.java:78)
>>> Caused by: java.lang.IllegalArgumentException: Column not found by
>>> KYLIN_SALES.USER_ID
>>> at org.apache.kylin.metadata.model.DataModelDesc.findColumn(Dat
>>> aModelDesc.java:240)
>>> at org.apache.kylin.metadata.model.ModelDimensionDesc.init(Mode
>>> lDimensionDesc.java:66)
>>> at org.apache.kylin.metadata.model.DataModelDesc.initDimensions
>>> AndMetrics(DataModelDesc.java:381)
>>> at org.apache.kylin.metadata.model.DataModelDesc.init(DataModel
>>> Desc.java:302)
>>> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDes
>>> cAt(MetadataManager.java:554)
>>> ... 9 more
>>> 2017-07-03 16:33:36,194 DEBUG [main MetadataManager:543]: Loaded 8
>>> DataModel(s)
>>> ……
>>>
>>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>>> table OK
>>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>>> table Time taken: 1.566 seconds
>>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>>> table OK
>>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>>> table kylin_intermediate_cube_dim_order_one_6b1de133_a10b_4bb6_86a
>>> 7_f249e6d0c418
>>> Exception in thread "main" java.lang.RuntimeException: error execute
>>> org.apache.kylin.storage.hbase.util.StorageCleanupJob
>>> at org.apache.kylin.common.util.AbstractApplication.execute(Abs
>>> tractApplication.java:42)
>>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(S
>>> torageCleanupJob.java:362)
>>> Caused by: java.lang.NullPointerException
>>> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(Class
>>> Util.java:84)
>>> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.jav
>>> a:76)
>>> at org.apache.kylin.job.execution.ExecutableManager.parseTo(Exe
>>> cutableManager.java:439)
>>> at org.apache.kylin.job.execution.ExecutableManager.getJob(Exec
>>> utableManager.java:129)
>>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTabl
>>> eInUse(StorageCleanupJob.java:347)
>>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanU
>>> nusedIntermediateHiveTable(StorageCleanupJob.java:302)
>>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execut
>>> e(StorageCleanupJob.java:153)
>>> at org.apache.kylin.common.util.AbstractApplication.execute(Abs
>>> tractApplication.java:37)
>>> ... 1 more
>>> 2017-07-03 16:33:43,342 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2259]:
>>> Closing master protocol: MasterService
>>> 2017-07-03 16:33:43,368 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1830]:
>>> Closing zookeeper sessionid=0x25cbe7e555d9dc5
>>> 2017-07-03 16:33:43,371 INFO [Thread-0 ZooKeeper:684]: Session:
>>> 0x25cbe7e555d9dc5 closed
>>> 2017-07-03 16:33:43,371 INFO [main-EventThread ClientCnxn:512]:
>>> EventThread shut down
>>>
>>> Best regards.
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
Re: StorageCleanup error
Posted by ZhangXuan <zh...@czkj1010.com>.
I build new version and test it ,but the error still exist, so I use 1.6 to clean up first, and next run in 2.0.0, error disappeared.
Maybe the NullPointerException bug did not fixed in new version.
Thanks
> 在 2017年7月4日,09:20,ShaoFeng Shi <sh...@apache.org> 写道:
>
> It is easy, you can follow the doc to build a new package: https://kylin.apache.org/development/howto_package.html <https://kylin.apache.org/development/howto_package.html>
>
>
> 2017-07-03 22:53 GMT+08:00 张绚 <zhx@czkj1010.com <ma...@czkj1010.com>>:
> Hi,
> I has read it ,but 2.0.0 is the last release version on website, someone can give me new version package or jar files. Thanks.
>
>> 在 2017年7月3日,22:32,ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org>> 写道:
>>
>> Hi xuan,
>>
>> Check this:
>>
>> http://apache-kylin.74782.x6.nabble.com/errors-when-run-StorageCleanupJob-td8321.html <http://apache-kylin.74782.x6.nabble.com/errors-when-run-StorageCleanupJob-td8321.html>
>>
>> 2017-07-03 17:01 GMT+08:00 张绚 <zhx@czkj1010.com <ma...@czkj1010.com>>:
>> Hi
>> My test hadoop storage is almost full, so I use StorageCleanupJob to clear the unused data, but got the following error, please help analyze it and give me some suggestions, My kylin has upgraded from 1.6 to 2.0 now. Thanks.
>>
>> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:384]: Loaded 36 SourceTable EXD(s)
>> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:529]: Reloading DataModel from folder kylin_metadata(key='/model_desc')@kylin_metadata@hbase
>> 2017-07-03 16:33:36,184 ERROR [main MetadataManager:538]: Error to load DataModel at /model_desc/test_model.json
>> java.lang.IllegalStateException: Error to load /model_desc/test_model.json
>> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:558)
>> at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.java:536)
>> at org.apache.kylin.metadata.MetadataManager.init(MetadataManager.java:284)
>> at org.apache.kylin.metadata.MetadataManager.<init>(MetadataManager.java:113)
>> at org.apache.kylin.metadata.MetadataManager.getInstance(MetadataManager.java:83)
>> at org.apache.kylin.job.dao.ExecutableDao.<init>(ExecutableDao.java:76)
>> at org.apache.kylin.job.dao.ExecutableDao.getInstance(ExecutableDao.java:57)
>> at org.apache.kylin.job.execution.ExecutableManager.<init>(ExecutableManager.java:55)
>> at org.apache.kylin.job.execution.ExecutableManager.getInstance(ExecutableManager.java:64)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.<clinit>(StorageCleanupJob.java:78)
>> Caused by: java.lang.IllegalArgumentException: Column not found by KYLIN_SALES.USER_ID
>> at org.apache.kylin.metadata.model.DataModelDesc.findColumn(DataModelDesc.java:240)
>> at org.apache.kylin.metadata.model.ModelDimensionDesc.init(ModelDimensionDesc.java:66)
>> at org.apache.kylin.metadata.model.DataModelDesc.initDimensionsAndMetrics(DataModelDesc.java:381)
>> at org.apache.kylin.metadata.model.DataModelDesc.init(DataModelDesc.java:302)
>> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:554)
>> ... 9 more
>> 2017-07-03 16:33:36,194 DEBUG [main MetadataManager:543]: Loaded 8 DataModel(s)
>> ……
>>
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table OK
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table Time taken: 1.566 seconds
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table OK
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_dim_order_one_6b1de133_a10b_4bb6_86a7_f249e6d0c418
>> Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.util.StorageCleanupJob
>> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:362)
>> Caused by: java.lang.NullPointerException
>> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:84)
>> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76)
>> at org.apache.kylin.job.execution.ExecutableManager.parseTo(ExecutableManager.java:439)
>> at org.apache.kylin.job.execution.ExecutableManager.getJob(ExecutableManager.java:129)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(StorageCleanupJob.java:347)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(StorageCleanupJob.java:153)
>> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
>> ... 1 more
>> 2017-07-03 16:33:43,342 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2259]: Closing master protocol: MasterService
>> 2017-07-03 16:33:43,368 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1830]: Closing zookeeper sessionid=0x25cbe7e555d9dc5
>> 2017-07-03 16:33:43,371 INFO [Thread-0 ZooKeeper:684]: Session: 0x25cbe7e555d9dc5 closed
>> 2017-07-03 16:33:43,371 INFO [main-EventThread ClientCnxn:512]: EventThread shut down
>>
>> Best regards.
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: StorageCleanup error
Posted by ShaoFeng Shi <sh...@apache.org>.
It is easy, you can follow the doc to build a new package:
https://kylin.apache.org/development/howto_package.html
2017-07-03 22:53 GMT+08:00 张绚 <zh...@czkj1010.com>:
> Hi,
> I has read it ,but 2.0.0 is the last release version on website,
> someone can give me new version package or jar files. Thanks.
>
> 在 2017年7月3日,22:32,ShaoFeng Shi <sh...@apache.org> 写道:
>
> Hi xuan,
>
> Check this:
>
> http://apache-kylin.74782.x6.nabble.com/errors-when-run-
> StorageCleanupJob-td8321.html
>
> 2017-07-03 17:01 GMT+08:00 张绚 <zh...@czkj1010.com>:
>
>> Hi
>> My test hadoop storage is almost full, so I use StorageCleanupJob to
>> clear the unused data, but got the following error, please help analyze it
>> and give me some suggestions, My kylin has upgraded from 1.6 to 2.0 now.
>> Thanks.
>>
>> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:384]: Loaded 36
>> SourceTable EXD(s)
>> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:529]: Reloading
>> DataModel from folder kylin_metadata(key='/model_des
>> c')@kylin_metadata@hbase
>> 2017-07-03 16:33:36,184 ERROR [main MetadataManager:538]: Error to load
>> DataModel at /model_desc/test_model.json
>> java.lang.IllegalStateException: Error to load
>> /model_desc/test_model.json
>> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDes
>> cAt(MetadataManager.java:558)
>> at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel
>> (MetadataManager.java:536)
>> at org.apache.kylin.metadata.MetadataManager.init(MetadataManag
>> er.java:284)
>> at org.apache.kylin.metadata.MetadataManager.<init>(MetadataMan
>> ager.java:113)
>> at org.apache.kylin.metadata.MetadataManager.getInstance(Metada
>> taManager.java:83)
>> at org.apache.kylin.job.dao.ExecutableDao.<init>(ExecutableDao.
>> java:76)
>> at org.apache.kylin.job.dao.ExecutableDao.getInstance(Executabl
>> eDao.java:57)
>> at org.apache.kylin.job.execution.ExecutableManager.<init>(
>> ExecutableManager.java:55)
>> at org.apache.kylin.job.execution.ExecutableManager.getInstance
>> (ExecutableManager.java:64)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.<clini
>> t>(StorageCleanupJob.java:78)
>> Caused by: java.lang.IllegalArgumentException: Column not found by
>> KYLIN_SALES.USER_ID
>> at org.apache.kylin.metadata.model.DataModelDesc.findColumn(
>> DataModelDesc.java:240)
>> at org.apache.kylin.metadata.model.ModelDimensionDesc.init(Mode
>> lDimensionDesc.java:66)
>> at org.apache.kylin.metadata.model.DataModelDesc.initDimensions
>> AndMetrics(DataModelDesc.java:381)
>> at org.apache.kylin.metadata.model.DataModelDesc.init(DataModel
>> Desc.java:302)
>> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDes
>> cAt(MetadataManager.java:554)
>> ... 9 more
>> 2017-07-03 16:33:36,194 DEBUG [main MetadataManager:543]: Loaded 8
>> DataModel(s)
>> ……
>>
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>> table OK
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>> table Time taken: 1.566 seconds
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>> table OK
>> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking
>> table kylin_intermediate_cube_dim_order_one_6b1de133_a10b_4bb6_86a
>> 7_f249e6d0c418
>> Exception in thread "main" java.lang.RuntimeException: error execute
>> org.apache.kylin.storage.hbase.util.StorageCleanupJob
>> at org.apache.kylin.common.util.AbstractApplication.execute(Abs
>> tractApplication.java:42)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(
>> StorageCleanupJob.java:362)
>> Caused by: java.lang.NullPointerException
>> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(Class
>> Util.java:84)
>> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.jav
>> a:76)
>> at org.apache.kylin.job.execution.ExecutableManager.parseTo(
>> ExecutableManager.java:439)
>> at org.apache.kylin.job.execution.ExecutableManager.getJob(
>> ExecutableManager.java:129)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTabl
>> eInUse(StorageCleanupJob.java:347)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanU
>> nusedIntermediateHiveTable(StorageCleanupJob.java:302)
>> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execut
>> e(StorageCleanupJob.java:153)
>> at org.apache.kylin.common.util.AbstractApplication.execute(Abs
>> tractApplication.java:37)
>> ... 1 more
>> 2017-07-03 16:33:43,342 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2259]:
>> Closing master protocol: MasterService
>> 2017-07-03 16:33:43,368 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1830]:
>> Closing zookeeper sessionid=0x25cbe7e555d9dc5
>> 2017-07-03 16:33:43,371 INFO [Thread-0 ZooKeeper:684]: Session:
>> 0x25cbe7e555d9dc5 closed
>> 2017-07-03 16:33:43,371 INFO [main-EventThread ClientCnxn:512]:
>> EventThread shut down
>>
>> Best regards.
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: StorageCleanup error
Posted by 张绚 <zh...@czkj1010.com>.
Hi,
I has read it ,but 2.0.0 is the last release version on website, someone can give me new version package or jar files. Thanks.
> 在 2017年7月3日,22:32,ShaoFeng Shi <sh...@apache.org> 写道:
>
> Hi xuan,
>
> Check this:
>
> http://apache-kylin.74782.x6.nabble.com/errors-when-run-StorageCleanupJob-td8321.html <http://apache-kylin.74782.x6.nabble.com/errors-when-run-StorageCleanupJob-td8321.html>
>
> 2017-07-03 17:01 GMT+08:00 张绚 <zhx@czkj1010.com <ma...@czkj1010.com>>:
> Hi
> My test hadoop storage is almost full, so I use StorageCleanupJob to clear the unused data, but got the following error, please help analyze it and give me some suggestions, My kylin has upgraded from 1.6 to 2.0 now. Thanks.
>
> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:384]: Loaded 36 SourceTable EXD(s)
> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:529]: Reloading DataModel from folder kylin_metadata(key='/model_desc')@kylin_metadata@hbase
> 2017-07-03 16:33:36,184 ERROR [main MetadataManager:538]: Error to load DataModel at /model_desc/test_model.json
> java.lang.IllegalStateException: Error to load /model_desc/test_model.json
> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:558)
> at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.java:536)
> at org.apache.kylin.metadata.MetadataManager.init(MetadataManager.java:284)
> at org.apache.kylin.metadata.MetadataManager.<init>(MetadataManager.java:113)
> at org.apache.kylin.metadata.MetadataManager.getInstance(MetadataManager.java:83)
> at org.apache.kylin.job.dao.ExecutableDao.<init>(ExecutableDao.java:76)
> at org.apache.kylin.job.dao.ExecutableDao.getInstance(ExecutableDao.java:57)
> at org.apache.kylin.job.execution.ExecutableManager.<init>(ExecutableManager.java:55)
> at org.apache.kylin.job.execution.ExecutableManager.getInstance(ExecutableManager.java:64)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.<clinit>(StorageCleanupJob.java:78)
> Caused by: java.lang.IllegalArgumentException: Column not found by KYLIN_SALES.USER_ID
> at org.apache.kylin.metadata.model.DataModelDesc.findColumn(DataModelDesc.java:240)
> at org.apache.kylin.metadata.model.ModelDimensionDesc.init(ModelDimensionDesc.java:66)
> at org.apache.kylin.metadata.model.DataModelDesc.initDimensionsAndMetrics(DataModelDesc.java:381)
> at org.apache.kylin.metadata.model.DataModelDesc.init(DataModelDesc.java:302)
> at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:554)
> ... 9 more
> 2017-07-03 16:33:36,194 DEBUG [main MetadataManager:543]: Loaded 8 DataModel(s)
> ……
>
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table OK
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table Time taken: 1.566 seconds
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table OK
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_dim_order_one_6b1de133_a10b_4bb6_86a7_f249e6d0c418
> Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.util.StorageCleanupJob
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:362)
> Caused by: java.lang.NullPointerException
> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:84)
> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76)
> at org.apache.kylin.job.execution.ExecutableManager.parseTo(ExecutableManager.java:439)
> at org.apache.kylin.job.execution.ExecutableManager.getJob(ExecutableManager.java:129)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(StorageCleanupJob.java:347)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(StorageCleanupJob.java:153)
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 1 more
> 2017-07-03 16:33:43,342 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2259]: Closing master protocol: MasterService
> 2017-07-03 16:33:43,368 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1830]: Closing zookeeper sessionid=0x25cbe7e555d9dc5
> 2017-07-03 16:33:43,371 INFO [Thread-0 ZooKeeper:684]: Session: 0x25cbe7e555d9dc5 closed
> 2017-07-03 16:33:43,371 INFO [main-EventThread ClientCnxn:512]: EventThread shut down
>
> Best regards.
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: StorageCleanup error
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi xuan,
Check this:
http://apache-kylin.74782.x6.nabble.com/errors-when-run-StorageCleanupJob-td8321.html
2017-07-03 17:01 GMT+08:00 张绚 <zh...@czkj1010.com>:
> Hi
> My test hadoop storage is almost full, so I use StorageCleanupJob to
> clear the unused data, but got the following error, please help analyze it
> and give me some suggestions, My kylin has upgraded from 1.6 to 2.0 now.
> Thanks.
>
> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:384]: Loaded 36
> SourceTable EXD(s)
> 2017-07-03 16:33:36,111 DEBUG [main MetadataManager:529]: Reloading
> DataModel from folder kylin_metadata(key='/model_
> desc')@kylin_metadata@hbase
> 2017-07-03 16:33:36,184 ERROR [main MetadataManager:538]: Error to load
> DataModel at /model_desc/test_model.json
> java.lang.IllegalStateException: Error to load /model_desc/test_model.json
> at org.apache.kylin.metadata.MetadataManager.
> reloadDataModelDescAt(MetadataManager.java:558)
> at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(
> MetadataManager.java:536)
> at org.apache.kylin.metadata.MetadataManager.init(
> MetadataManager.java:284)
> at org.apache.kylin.metadata.MetadataManager.<init>(
> MetadataManager.java:113)
> at org.apache.kylin.metadata.MetadataManager.getInstance(
> MetadataManager.java:83)
> at org.apache.kylin.job.dao.ExecutableDao.<init>(
> ExecutableDao.java:76)
> at org.apache.kylin.job.dao.ExecutableDao.getInstance(
> ExecutableDao.java:57)
> at org.apache.kylin.job.execution.ExecutableManager.<
> init>(ExecutableManager.java:55)
> at org.apache.kylin.job.execution.ExecutableManager.
> getInstance(ExecutableManager.java:64)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.<
> clinit>(StorageCleanupJob.java:78)
> Caused by: java.lang.IllegalArgumentException: Column not found by
> KYLIN_SALES.USER_ID
> at org.apache.kylin.metadata.model.DataModelDesc.
> findColumn(DataModelDesc.java:240)
> at org.apache.kylin.metadata.model.ModelDimensionDesc.init(
> ModelDimensionDesc.java:66)
> at org.apache.kylin.metadata.model.DataModelDesc.
> initDimensionsAndMetrics(DataModelDesc.java:381)
> at org.apache.kylin.metadata.model.DataModelDesc.init(
> DataModelDesc.java:302)
> at org.apache.kylin.metadata.MetadataManager.
> reloadDataModelDescAt(MetadataManager.java:554)
> ... 9 more
> 2017-07-03 16:33:36,194 DEBUG [main MetadataManager:543]: Loaded 8
> DataModel(s)
> ……
>
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table
> OK
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table
> Time taken: 1.566 seconds
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table
> OK
> 2017-07-03 16:33:43,335 INFO [main StorageCleanupJob:283]: Checking table
> kylin_intermediate_cube_dim_order_one_6b1de133_a10b_4bb6_86a7_f249e6d0c418
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.util.StorageCleanupJob
> at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:42)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> main(StorageCleanupJob.java:362)
> Caused by: java.lang.NullPointerException
> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(
> ClassUtil.java:84)
> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.
> java:76)
> at org.apache.kylin.job.execution.ExecutableManager.
> parseTo(ExecutableManager.java:439)
> at org.apache.kylin.job.execution.ExecutableManager.
> getJob(ExecutableManager.java:129)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> isTableInUse(StorageCleanupJob.java:347)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> execute(StorageCleanupJob.java:153)
> at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
> ... 1 more
> 2017-07-03 16:33:43,342 INFO [Thread-0 ConnectionManager$
> HConnectionImplementation:2259]: Closing master protocol: MasterService
> 2017-07-03 16:33:43,368 INFO [Thread-0 ConnectionManager$
> HConnectionImplementation:1830]: Closing zookeeper
> sessionid=0x25cbe7e555d9dc5
> 2017-07-03 16:33:43,371 INFO [Thread-0 ZooKeeper:684]: Session:
> 0x25cbe7e555d9dc5 closed
> 2017-07-03 16:33:43,371 INFO [main-EventThread ClientCnxn:512]:
> EventThread shut down
>
> Best regards.
>
>
--
Best regards,
Shaofeng Shi 史少锋