You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Fei Yi <yi...@gmail.com> on 2017/09/16 05:20:51 UTC
Load HFile to HBase Table Error
2017-09-16 12:26:02,431 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641]
mapreduce.LoadIncrementalHFiles:439 : Split occured while grouping HFiles,
retry attempt 10 with 18 files remaining to group or split
2017-09-16 12:26:02,449 ERROR [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641]
mapreduce.LoadIncrementalHFiles:482 :
-------------------------------------------------
Bulk load aborted with some files not yet loaded:
-------------------------------------------------
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/3582d9fa7e2b4f528a3f3faa0f7fd7bd
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/738feed13b9a43f9a34ab2d2e7cf6b03
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/be0b4e45309a4ca49ddc2aa7402f408f
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/7dd4e4de28d9490ca249b69b216fae89
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/f5802d050b4649b2895b8c780825fd70
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ccf41c1f2d78436db3d772bc5facc415
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/eb2dc01f7ee84ebb9d913ef3627dbea2
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/e0a0d08920e54614b1515a1a698dfe81
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/285323e81d5c49a8acb45bb5e30e5d98
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/e91dda576b1c4d29a9e0c0f39a0aca97
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/10fea9b19da344e4a28d7d3148715593
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/1aaf33082e93469eb420c61af4b41561
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ef09b5baeefe4e0aac40b20d61ecc720
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/2f3a9d028e61474e8b191278cdb448f9
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/c0b0944aee544f86801711fb36b49a16
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/c7cd732f1b6f436abf439c13d2c7e111
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/2c310e63ed074d378b82c807bdcc0d65
hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/e70e10ed69534ff0a1081db4775fcf0f
2017-09-16 12:26:02,449 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641]
client.ConnectionManager$HConnectionImplementation:2259 : Closing master
protocol: MasterService
2017-09-16 12:26:02,449 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641]
client.ConnectionManager$HConnectionImplementation:1830 : Closing zookeeper
sessionid=0x25e31997e98c034
2017-09-16 12:26:02,461 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641] zookeeper.ZooKeeper:684 :
Session: 0x25e31997e98c034 closed
2017-09-16 12:26:02,461 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641-EventThread]
zookeeper.ClientCnxn:512 : EventThread shut down
2017-09-16 12:26:02,472 ERROR [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641] common.HadoopShellExecutable:64
: error execute
HadoopShellExecutable{id=fffd693d-dfcd-4235-82e0-5ede25687805-13, name=Load
HFile to HBase Table, state=RUNNING}
java.io.IOException: Retry attempted 10 times without completing, bailing
out
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:446)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:339)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1063)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.storage.hbase.steps.BulkLoadJob.run(BulkLoadJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:65)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:141)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-09-16 12:26:02,479 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641] execution.ExecutableManager:425
: job id:fffd693d-dfcd-4235-82e0-5ede25687805-13 from RUNNING to ERROR
2017-09-16 12:26:02,499 INFO [Job
fffd693d-dfcd-4235-82e0-5ede25687805-4641] execution.ExecutableManager:425
: job id:fffd693d-dfcd-4235-82e0-5ede25687805 from RUNNING to ERROR
Re: Load HFile to HBase Table Error
Posted by ShaoFeng Shi <sh...@apache.org>.
Could you provide more information like Kylin/Hadoop version, your
deployment topology, how do you encounter this problem (what changes made
before getting this problem), etc? And what findings you have now?
2017-09-16 13:20 GMT+08:00 Fei Yi <yi...@gmail.com>:
>
>
> 2017-09-16 12:26:02,431 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> mapreduce.LoadIncrementalHFiles:439 : Split occured while grouping
> HFiles, retry attempt 10 with 18 files remaining to group or split
> 2017-09-16 12:26:02,449 ERROR [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> mapreduce.LoadIncrementalHFiles:482 : ------------------------------
> -------------------
> Bulk load aborted with some files not yet loaded:
> -------------------------------------------------
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> 3582d9fa7e2b4f528a3f3faa0f7fd7bd
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> 738feed13b9a43f9a34ab2d2e7cf6b03
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> be0b4e45309a4ca49ddc2aa7402f408f
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> 7dd4e4de28d9490ca249b69b216fae89
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> f5802d050b4649b2895b8c780825fd70
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> ccf41c1f2d78436db3d772bc5facc415
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> eb2dc01f7ee84ebb9d913ef3627dbea2
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> e0a0d08920e54614b1515a1a698dfe81
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> 285323e81d5c49a8acb45bb5e30e5d98
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> e91dda576b1c4d29a9e0c0f39a0aca97
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> 10fea9b19da344e4a28d7d3148715593
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> 1aaf33082e93469eb420c61af4b41561
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> ef09b5baeefe4e0aac40b20d61ecc720
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> 2f3a9d028e61474e8b191278cdb448f9
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> c0b0944aee544f86801711fb36b49a16
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/
> c7cd732f1b6f436abf439c13d2c7e111
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> 2c310e63ed074d378b82c807bdcc0d65
> hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d-
> dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/
> e70e10ed69534ff0a1081db4775fcf0f
>
> 2017-09-16 12:26:02,449 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> client.ConnectionManager$HConnectionImplementation:2259 : Closing master
> protocol: MasterService
> 2017-09-16 12:26:02,449 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> client.ConnectionManager$HConnectionImplementation:1830 : Closing
> zookeeper sessionid=0x25e31997e98c034
> 2017-09-16 12:26:02,461 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> zookeeper.ZooKeeper:684 : Session: 0x25e31997e98c034 closed
> 2017-09-16 12:26:02,461 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641-EventThread]
> zookeeper.ClientCnxn:512 : EventThread shut down
> 2017-09-16 12:26:02,472 ERROR [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> common.HadoopShellExecutable:64 : error execute HadoopShellExecutable{id=
> fffd693d-dfcd-4235-82e0-5ede25687805-13, name=Load HFile to HBase Table,
> state=RUNNING}
> java.io.IOException: Retry attempted 10 times without completing, bailing
> out
> at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.
> doBulkLoad(LoadIncrementalHFiles.java:446)
> at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.
> doBulkLoad(LoadIncrementalHFiles.java:339)
> at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(
> LoadIncrementalHFiles.java:1063)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.kylin.storage.hbase.steps.BulkLoadJob.run(
> BulkLoadJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.kylin.engine.mr.common.HadoopShellExecutable.
> doWork(HadoopShellExecutable.java:62)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:125)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:65)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:125)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:141)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-16 12:26:02,479 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> execution.ExecutableManager:425 : job id:fffd693d-dfcd-4235-82e0-5ede25687805-13
> from RUNNING to ERROR
> 2017-09-16 12:26:02,499 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641]
> execution.ExecutableManager:425 : job id:fffd693d-dfcd-4235-82e0-5ede25687805
> from RUNNING to ERROR
>
>
--
Best regards,
Shaofeng Shi 史少锋