You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Billy Liu <bi...@apache.org> on 2017/07/24 14:02:23 UTC
Re: error execute org.apache.kylin.storage. hbase.util.StorageCleanupJob
I think the issue has been fixed in latest master code. Could you pull the
code and have a try?
2017-07-24 18:24 GMT+08:00 liulang <li...@mobiexchanger.com>:
> hi，
> Run StorageCleanupJob error：
>
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Class path contains multiple SLF4J bindings.
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/apache-
> hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/
> slf4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/apache-
> kylin-2.0.0-bin/spark/lib/spark-assembly-1.6.3-hadoop2.
> 6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/tez-0.8.4/
> lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.
> 7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/
> slf4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.
> Log4jLoggerFactory]
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> Logging initialized using configuration in jar:file:/opt/apps/apache-
> hive-2.0.1-bin/lib/hive-common-2.0.1.jar!/hive-log4j2.properties
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> OK
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> Time taken: 0.924 seconds
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> OK
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> kylin_intermediate_cube_dsp_report_f92212f6_738d_4ef2_a727_1796e30c6cb0
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.util.StorageCleanupJob
> at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:42)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> main(StorageCleanupJob.java:362)
> Caused by: java.lang.NullPointerException
> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(
> ClassUtil.java:84)
> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76)
> at org.apache.kylin.job.execution.ExecutableManager.
> parseTo(ExecutableManager.java:439)
> at org.apache.kylin.job.execution.ExecutableManager.
> getJob(ExecutableManager.java:129)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(
> StorageCleanupJob.java:347)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> execute(StorageCleanupJob.java:153)
> at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
> ... 1 more
> 2017-07-24 16:04:28,652 INFO [Thread-0 ConnectionManager$
> HConnectionImplementation:2068]: Closing master protocol: MasterService
> 2017-07-24 16:04:28,656 INFO [Thread-0 ConnectionManager$
> HConnectionImplementation:1676]: Closing zookeeper
> sessionid=0x15c4288a5da15f6
> 2017-07-24 16:04:28,683 INFO [Thread-0 ZooKeeper:684]: Session:
> 0x15c4288a5da15f6 closed
> 2017-07-24 16:04:28,683 INFO [main-EventThread ClientCnxn:512]:
> EventThread shut down
>
> Thanks。
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: error execute org.apache.kylin.storage. hbase.util.StorageCleanupJob
Posted by Li Yang <li...@apache.org>.
Note "org.apache.kylin.storage.hbase.util.StorageCleanupJob" is deprecated.
Please use "org.apache.kylin.tool.StorageCleanupJob" instead.
I'm updating related document.
Thanks for reporting!
Yang
On Mon, Jul 31, 2017 at 6:35 PM, liulang <li...@mobiexchanger.com> wrote:
> Hi，重新打包了最新源码，还是这样；
> 并且我按步骤
> Step1: run command 'bin/kylin.sh org.apache.kylin.tool.AclTableMigrationCLI
> MIGRATE’, 这个已执行；
> Step2: drop hbase tables: kylin_metadata_acl and kylin_metadata_user
> 这个没敢执行，怕影响2.0的版本使用；
>
> 异常信息：
>
> 2017-07-31 18:26:20,159 INFO [main StorageCleanupJob:283]: Checking table
> kylin_intermediate_cube_dsp_reportnbr_0e935361_9670_4d66_952a_a597abcc730f
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.util.StorageCleanupJob
> at org.apache.kylin.common.util.AbstractApplication.execute(Ab
> stractApplication.java:42 <http://abstractapplication.java:42>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(
> StorageCleanupJob.java:362 <http://storagecleanupjob.java:362>)
> Caused by: java.lang.NullPointerException
> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(Clas
> sUtil.java:86 <http://classutil.java:86>)
> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:78
> <http://classutil.java:78>)
> at org.apache.kylin.job.execution.ExecutableManager.parseTo(
> ExecutableManager.java:494 <http://executablemanager.java:494>)
> at org.apache.kylin.job.execution.ExecutableManager.getJob(
> ExecutableManager.java:141 <http://executablemanager.java:141>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(
> StorageCleanupJob.java:347 <http://storagecleanupjob.java:347>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.
> cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302
> <http://storagecleanupjob.java:302>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(
> StorageCleanupJob.java:153 <http://storagecleanupjob.java:153>)
> at org.apache.kylin.common.util.AbstractApplication.execute(Ab
> stractApplication.java:37 <http://abstractapplication.java:37>)
> ... 1 more
> 2017-07-31 18:26:20,165 INFO [Thread-1 ConnectionManager$
> HConnectionImplementation:2068]: Closing master protocol: MasterService
> 2017-07-31 18:26:20,173 INFO [Thread-1 ConnectionManager$
> HConnectionImplementation:1676]: Closing zookeeper
> sessionid=0x25c4288a4a417e4
> 2017-07-31 18:26:20,175 INFO [Thread-1 ZooKeeper:684]: Session:
> 0x25c4288a4a417e4 closed
> 2017-07-31 18:26:20,175 INFO [main-EventThread ClientCnxn:512]:
> EventThread shut down
>
>
> 以上，请指导，谢谢。
> < Billy Liu <bi...@apache.org>> 在 2017-07-24 22:02:23 写道:
>
> I think the issue has been fixed in latest master code. Could you pull the
> code and have a try?
>
> 2017-07-24 18:24 GMT+08:00 liulang <li...@mobiexchanger.com>:
>
> hi，
> Run StorageCleanupJob error：
>
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Class path contains multiple SLF4J bindings.
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/apache-hiv
> e-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/apache-kyl
> in-2.0.0-bin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.
> jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/tez-0.8.4/
> lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.7
> .2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf
> 4j/impl/StaticLoggerBinder.class]
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4
> jLoggerFactory]
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> Logging initialized using configuration in jar:file:/opt/apps/apache-hive
> -2.0.1-bin/lib/hive-common-2.0.1.jar!/hive-log4j2.properties
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> OK
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> Time taken: 0.924 seconds
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> OK
> 2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
> kylin_intermediate_cube_dsp_report_f92212f6_738d_4ef2_a727_1796e30c6cb0
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.util.StorageCleanupJob
> at org.apache.kylin.common.util.AbstractApplication.execute(Abs
> tractApplication.java:42 <http://abstractapplication.java:42>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(S
> torageCleanupJob.java <http://storagecleanupjob.java>:362)
> Caused by: java.lang.NullPointerException
> at org.apache.kylin.common.util.ClassUtil.forRenamedClass(Class
> Util.java:84 <http://classutil.java:84>)
> at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76)
> at org.apache.kylin.job.execution.ExecutableManager.parseTo(
> ExecutableManager.java:439)
> at org.apache.kylin.job.execution.ExecutableManager.getJob(Exec
> utableManager.java <http://executablemanager.java>:129)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(
> StorageCleanupJob.java:347 <http://storagecleanupjob.java:347>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanU
> nusedIntermediateHiveTable(StorageCleanupJob.java:302
> <http://storagecleanupjob.java:302>)
> at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execut
> e(StorageCleanupJob.java:153)
> at org.apache.kylin.common.util.AbstractApplication.execute(Abs
> tractApplication.java:37 <http://abstractapplication.java:37>)
> ... 1 more
> 2017-07-24 16:04:28,652 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2068]:
> Closing master protocol: MasterService
> 2017-07-24 16:04:28,656 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1676]:
> Closing zookeeper sessionid=0x15c4288a5da15f6
> 2017-07-24 16:04:28,683 INFO [Thread-0 ZooKeeper:684]: Session:
> 0x15c4288a5da15f6 closed
> 2017-07-24 16:04:28,683 INFO [main-EventThread ClientCnxn:512]:
> EventThread shut down
>
> Thanks。
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: error execute org.apache.kylin.storage. hbase.util.StorageCleanupJob
Posted by liulang <li...@mobiexchanger.com>.
Hi，重新打包了最新源码，还是这样；
并且我按步骤
Step1: run command 'bin/kylin.sh org.apache.kylin.tool.AclTableMigrationCLI MIGRATE’, 这个已执行；
Step2: drop hbase tables: kylin_metadata_acl and kylin_metadata_user 这个没敢执行，怕影响2.0的版本使用；
异常信息：
2017-07-31 18:26:20,159 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_dsp_reportnbr_0e935361_9670_4d66_952a_a597abcc730f
Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.util.StorageCleanupJob
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:362)
Caused by: java.lang.NullPointerException
at org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:86)
at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:78)
at org.apache.kylin.job.execution.ExecutableManager.parseTo(ExecutableManager.java:494)
at org.apache.kylin.job.execution.ExecutableManager.getJob(ExecutableManager.java:141)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(StorageCleanupJob.java:347)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(StorageCleanupJob.java:153)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 1 more
2017-07-31 18:26:20,165 INFO [Thread-1 ConnectionManager$HConnectionImplementation:2068]: Closing master protocol: MasterService
2017-07-31 18:26:20,173 INFO [Thread-1 ConnectionManager$HConnectionImplementation:1676]: Closing zookeeper sessionid=0x25c4288a4a417e4
2017-07-31 18:26:20,175 INFO [Thread-1 ZooKeeper:684]: Session: 0x25c4288a4a417e4 closed
2017-07-31 18:26:20,175 INFO [main-EventThread ClientCnxn:512]: EventThread shut down
以上，请指导，谢谢。
< Billy Liu> 在 2017-07-24 22:02:23 写道:
I think the issue has been fixed in latest master code. Could you pull the code and have a try?
2017-07-24 18:24 GMT+08:00 liulang <li...@mobiexchanger.com>:
hi，
Run StorageCleanupJob error：
2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table SLF4J: Class path contains multiple SLF4J bindings.
2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table SLF4J: Found binding in [jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table SLF4J: Found binding in [jar:file:/opt/apps/apache-kylin-2.0.0-bin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table SLF4J: Found binding in [jar:file:/opt/apps/tez-0.8.4/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-07-24 16:04:28,645 INFO [main StorageCleanupJob:283]: Checking table SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table Logging initialized using configuration in jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/hive-common-2.0.1.jar!/hive-log4j2.properties
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table OK
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table Time taken: 0.924 seconds
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table OK
2017-07-24 16:04:28,646 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_dsp_report_f92212f6_738d_4ef2_a727_1796e30c6cb0
Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.util.StorageCleanupJob
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:362)
Caused by: java.lang.NullPointerException
at org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:84)
at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76)
at org.apache.kylin.job.execution.ExecutableManager.parseTo(ExecutableManager.java:439)
at org.apache.kylin.job.execution.ExecutableManager.getJob(ExecutableManager.java:129)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(StorageCleanupJob.java:347)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302)
at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(StorageCleanupJob.java:153)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 1 more
2017-07-24 16:04:28,652 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2068]: Closing master protocol: MasterService
2017-07-24 16:04:28,656 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1676]: Closing zookeeper sessionid=0x15c4288a5da15f6
2017-07-24 16:04:28,683 INFO [Thread-0 ZooKeeper:684]: Session: 0x15c4288a5da15f6 closed
2017-07-24 16:04:28,683 INFO [main-EventThread ClientCnxn:512]: EventThread shut down
Thanks。