You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by mnagy <ma...@quantium.com.au> on 2016/11/02 04:48:20 UTC
13 Step Name: Load HFile to HBase Table
I am facing this issue when I try to build a cube.
Any ideas about what is going on ?
java.io.IOException: BulkLoad encountered an unrecoverable problem
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:510)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:441)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:331)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1025)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.storage.hbase.steps.BulkLoadJob.run(BulkLoadJob.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:115)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:115)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=5, exceptions:
Wed Nov 02 12:25:15 GMT+08:00 2016,
RpcRetryingCaller{globalStartTime=1478060655121, pause=3000, retries=5},
java.io.IOException: Call to
qtausc-pphd0107.quantium.com.au.local/192.168.81.107:16020 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=9,
waitTime=60001, operationTimeout=60000 expired.
Wed Nov 02 12:26:24 GMT+08:00 2016,
RpcRetryingCaller{globalStartTime=1478060655121, pause=3000, retries=5},
java.io.IOException: Call to
qtausc-pphd0107.quantium.com.au.local/192.168.81.107:16020 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=11,
waitTime=60001, operationTimeout=60000 expired.
Wed Nov 02 12:26:39 GMT+08:00 2016,
RpcRetryingCaller{globalStartTime=1478060655121, pause=3000, retries=5},
java.io.FileNotFoundException: java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/645ac77fe9c5441b9941f31bcdb58e3a
at com.mapr.fs.MapRClientImpl.open(MapRClientImpl.java:236)
at com.mapr.fs.MapRFileSystem.open(MapRFileSystem.java:960)
at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:803)
at
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:98)
at
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:79)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:519)
at
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:718)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5094)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1835)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32207)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Wed Nov 02 12:27:09 GMT+08:00 2016,
RpcRetryingCaller{globalStartTime=1478060655121, pause=3000, retries=5},
java.io.FileNotFoundException: java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/645ac77fe9c5441b9941f31bcdb58e3a
at com.mapr.fs.MapRClientImpl.open(MapRClientImpl.java:236)
at com.mapr.fs.MapRFileSystem.open(MapRFileSystem.java:960)
at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:803)
at
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:98)
at
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:79)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:519)
at
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:718)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5094)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1835)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32207)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Wed Nov 02 12:28:10 GMT+08:00 2016,
RpcRetryingCaller{globalStartTime=1478060655121, pause=3000, retries=5},
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.io.MultipleIOException):
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/645ac77fe9c5441b9941f31bcdb58e3a,
java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/0427031b0fad4b68b7ca83b7635186b8,
java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/a538edeb6de1484bb6161d00c1e27a87]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5107)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1835)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32207)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:813)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:489)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:486)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.io.MultipleIOException):
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/645ac77fe9c5441b9941f31bcdb58e3a,
java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/0427031b0fad4b68b7ca83b7635186b8,
java.io.FileNotFoundException:
/kylin/kylin_metadata/kylin-78027250-cc2b-4174-82eb-df1a50aa3f0c/simple_fact_smpl_cube_clone4/hfile/F1/a538edeb6de1484bb6161d00c1e27a87]
at
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5107)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1835)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32207)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1208)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.bulkLoadHFile(ClientProtos.java:32663)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1601)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$4.call(LoadIncrementalHFiles.java:767)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$4.call(LoadIncrementalHFiles.java:756)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 7 more
result code:2
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 13 Step Name: Load HFile to HBase Table
Posted by Alberto Ramón <a....@gmail.com>.
Hello
As ShaoFeng Shi said, LoadIncrementalHFiles step, is only copy / move HFile
to another path (under HBase Path)
I the past a had a lot of problems like this: The user which generates
HFile can't write on HBase folder
SOL1: Check the user who build HFile, is a Yarn user or is other (like
root, for example).
SOL2: Find HBase path, and change write permission
Other Idea: build and cp HFile, is a M-R task, check YARN's Log if there is
some like this:
"Error in LoadIncrementalHFiles
*Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=hbase, access=WRITE,
inode="/hbase/data/default/dataImport_05":hdfs:hbase:drwxr-xr-x* "
2016-11-03 6:51 GMT+01:00 ShaoFeng Shi <sh...@apache.org>:
> Is there any error/warning messages in kylin.log?
>
> 2016-11-03 11:24 GMT+08:00 mnagy <ma...@quantium.com.au>:
>
> > I already tried that and I got the same error at the same step.
> > I already tried to build a new cube and I got the same error as well.
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6164.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: 13 Step Name: Load HFile to HBase Table
Posted by mnagy <ma...@quantium.com.au>.
the error that I got is did not changed
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6190.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 13 Step Name: Load HFile to HBase Table
Posted by ShaoFeng Shi <sh...@apache.org>.
Is there any error/warning messages in kylin.log?
2016-11-03 11:24 GMT+08:00 mnagy <ma...@quantium.com.au>:
> I already tried that and I got the same error at the same step.
> I already tried to build a new cube and I got the same error as well.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6164.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋
Re: 13 Step Name: Load HFile to HBase Table
Posted by mnagy <ma...@quantium.com.au>.
I already tried that and I got the same error at the same step.
I already tried to build a new cube and I got the same error as well.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6164.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 13 Step Name: Load HFile to HBase Table
Posted by ShaoFeng Shi <sh...@apache.org>.
Now it is an inconsistent state, you can discard the current error job and
then re-run the build;
2016-11-03 6:25 GMT+08:00 mnagy <ma...@quantium.com.au>:
> Yes the table exist and is not empty.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6162.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋
Re: 13 Step Name: Load HFile to HBase Table
Posted by mnagy <ma...@quantium.com.au>.
Yes the table exist and is not empty.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6162.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 13 Step Name: Load HFile to HBase Table
Posted by ShaoFeng Shi <sh...@apache.org>.
Is the bulkload completed? You can check this by check whether the HTable
is empty or not; I guess it was completed as once bulkload is finished, the
file will be moved from Kylin's HDFS folder to HBase's folder;
2016-11-02 14:02 GMT+08:00 mnagy <ma...@quantium.com.au>:
> This folder is empty
>
> /kylin/kylin_metadata/kylin-1ee8b8f5-6708-43db-8285-
> e70b9181ca79/fact_smpl_big_cube/hfile/F1/
>
> what could cause that ?
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6148.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋
Re: 13 Step Name: Load HFile to HBase Table
Posted by mnagy <ma...@quantium.com.au>.
This folder is empty
/kylin/kylin_metadata/kylin-1ee8b8f5-6708-43db-8285-e70b9181ca79/fact_smpl_big_cube/hfile/F1/
what could cause that ?
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6148.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: 13 Step Name: Load HFile to HBase Table
Posted by mnagy <ma...@quantium.com.au>.
I am using.
kylin 1.6
ubuntu 14.04
hbase 1.1.1
hive 1.2
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/13-Step-Name-Load-HFile-to-HBase-Table-tp6144p6145.html
Sent from the Apache Kylin mailing list archive at Nabble.com.