You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Vineet Mishra <cl...@gmail.com> on 2015/07/10 08:25:52 UTC
Kylin Cube Creation Fails at Load HFile to Hbase Table
Hi All,
Currently I have around 3 cubes to build, which some times fails at Second
Last Step of Cube creation i.e Load Hfile to Hbase Table with the exception
java.io.IOException: BulkLoad encountered an unrecoverable problem
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:381)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:310)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:896)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.job.hadoop.hbase.BulkLoadJob.run(BulkLoadJob.java:83)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=35, exceptions:
Thu Jul 09 19:38:48 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:38:49 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:38:49 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:38:50 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:38:52 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:38:56 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:39:06 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:39:16 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:39:26 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:39:36 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:39:57 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:40:17 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:40:37 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:40:57 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:41:17 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:41:37 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:41:57 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:42:17 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:42:37 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:42:57 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:43:17 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:43:37 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:43:57 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:44:18 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:44:38 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:44:58 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:45:18 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:45:38 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:45:58 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:46:18 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:46:38 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:46:58 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:47:18 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:47:38 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
Thu Jul 09 19:47:59 PDT 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
java.net.UnknownHostException: unknown host: prod-hadoop-data02
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:129)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:679)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:360)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
Caused by: java.net.UnknownHostException: unknown host: prod-hadoop-data02
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(RpcClient.java:385)
at
org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1530)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.bulkLoadHFile(ClientProtos.java:30340)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1605)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1585)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:634)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:623)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
... 8 more
No doubt, the exception clearly states that its not able to reach to the
mentioned host, but unfortunately the host-ip entry was missing in the
hosts file which I made sure to include in the hosts file.
Moreover yesterday after including the host entry I re-ran the job(still it
wasn't working with resume) and it worked as expected but today again its
falling at the same point.
Any helpful advice would be appreciated.
Thanks,
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Li Yang <li...@apache.org>.
No. No other similar issue report.
On Mon, Jul 13, 2015 at 2:23 PM, Vineet Mishra <cl...@gmail.com>
wrote:
> Shi,
>
> I am using kylin-0.7.1, is this the version holding any such reported
> issues?
>
> Thanks,
>
> On Mon, Jul 13, 2015 at 8:49 AM, Shi, Shaofeng <sh...@ebay.com> wrote:
>
> > Vineet, I didn¹t see such error; Suggest you check Hbase logs for the
> root
> > cause; Kylin uses HBase¹s bulk load API to load data to Hbase; In our
> > deployment this step is quite stable, even for TB level cubes;
> >
> > On 7/12/15, 4:50 AM, "Vineet Mishra" <cl...@gmail.com> wrote:
> >
> > >Hi Li,
> > >
> > >I am not sure!
> > >
> > >As hadoop deployment is already running on amazon ec2 clusters which
> > >shouldn't be facing such issue.
> > >
> > >@Shi Any suggestions from your side?
> > >
> > >Thanks,
> > >
> > >On Sat, Jul 11, 2015 at 5:24 AM, Li Yang <li...@apache.org> wrote:
> > >
> > >> Unstable network? -- java.net.UnknownHostException: unknown host:
> > >> prod-hadoop-data02
> > >>
> > >>
> > >>
> > >> On Fri, Jul 10, 2015 at 7:15 PM, Sarnath <st...@gmail.com> wrote:
> > >>
> > >> > Disk space problem?
> > >> >
> > >>
> >
> >
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Vineet Mishra <cl...@gmail.com>.
Shi,
I am using kylin-0.7.1, is this the version holding any such reported
issues?
Thanks,
On Mon, Jul 13, 2015 at 8:49 AM, Shi, Shaofeng <sh...@ebay.com> wrote:
> Vineet, I didn¹t see such error; Suggest you check Hbase logs for the root
> cause; Kylin uses HBase¹s bulk load API to load data to Hbase; In our
> deployment this step is quite stable, even for TB level cubes;
>
> On 7/12/15, 4:50 AM, "Vineet Mishra" <cl...@gmail.com> wrote:
>
> >Hi Li,
> >
> >I am not sure!
> >
> >As hadoop deployment is already running on amazon ec2 clusters which
> >shouldn't be facing such issue.
> >
> >@Shi Any suggestions from your side?
> >
> >Thanks,
> >
> >On Sat, Jul 11, 2015 at 5:24 AM, Li Yang <li...@apache.org> wrote:
> >
> >> Unstable network? -- java.net.UnknownHostException: unknown host:
> >> prod-hadoop-data02
> >>
> >>
> >>
> >> On Fri, Jul 10, 2015 at 7:15 PM, Sarnath <st...@gmail.com> wrote:
> >>
> >> > Disk space problem?
> >> >
> >>
>
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
Vineet, I didn¹t see such error; Suggest you check Hbase logs for the root
cause; Kylin uses HBase¹s bulk load API to load data to Hbase; In our
deployment this step is quite stable, even for TB level cubes;
On 7/12/15, 4:50 AM, "Vineet Mishra" <cl...@gmail.com> wrote:
>Hi Li,
>
>I am not sure!
>
>As hadoop deployment is already running on amazon ec2 clusters which
>shouldn't be facing such issue.
>
>@Shi Any suggestions from your side?
>
>Thanks,
>
>On Sat, Jul 11, 2015 at 5:24 AM, Li Yang <li...@apache.org> wrote:
>
>> Unstable network? -- java.net.UnknownHostException: unknown host:
>> prod-hadoop-data02
>>
>>
>>
>> On Fri, Jul 10, 2015 at 7:15 PM, Sarnath <st...@gmail.com> wrote:
>>
>> > Disk space problem?
>> >
>>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Vineet Mishra <cl...@gmail.com>.
Hi Li,
I am not sure!
As hadoop deployment is already running on amazon ec2 clusters which
shouldn't be facing such issue.
@Shi Any suggestions from your side?
Thanks,
On Sat, Jul 11, 2015 at 5:24 AM, Li Yang <li...@apache.org> wrote:
> Unstable network? -- java.net.UnknownHostException: unknown host:
> prod-hadoop-data02
>
>
>
> On Fri, Jul 10, 2015 at 7:15 PM, Sarnath <st...@gmail.com> wrote:
>
> > Disk space problem?
> >
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Li Yang <li...@apache.org>.
Unstable network? -- java.net.UnknownHostException: unknown host:
prod-hadoop-data02
On Fri, Jul 10, 2015 at 7:15 PM, Sarnath <st...@gmail.com> wrote:
> Disk space problem?
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Vineet Mishra <cl...@gmail.com>.
Well Disk is not the problem since we have mounted around a TB of disk
space, which is having majority of free space!
On Fri, Jul 10, 2015 at 4:45 PM, Sarnath <st...@gmail.com> wrote:
> Disk space problem?
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Sarnath <st...@gmail.com>.
Disk space problem?
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Vineet Mishra <cl...@gmail.com>.
Moreover one more interesting thing which came to my mind was that since I
have 3 cubes in total and this is generally the issue I am facing while
building only one cube.
So basically 2/3 cubes build properly, but only one cube fails.
Let me mention it, the failing cubes are not same.
Thanks,
On Fri, Jul 10, 2015 at 11:55 AM, Vineet Mishra <cl...@gmail.com>
wrote:
> Hi All,
>
> Currently I have around 3 cubes to build, which some times fails at Second
> Last Step of Cube creation i.e Load Hfile to Hbase Table with the exception
>
> java.io.IOException: BulkLoad encountered an unrecoverable problem
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:381)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:310)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:896)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.kylin.job.hadoop.hbase.BulkLoadJob.run(BulkLoadJob.java:83)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=35, exceptions:
> Thu Jul 09 19:38:48 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:38:49 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:38:49 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:38:50 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:38:52 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:38:56 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:39:06 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:39:16 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:39:26 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:39:36 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:39:57 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:40:17 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:40:37 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:40:57 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:41:17 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:41:37 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:41:57 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:42:17 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:42:37 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:42:57 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:43:17 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:43:37 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:43:57 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:44:18 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:44:38 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:44:58 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:45:18 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:45:38 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:45:58 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:46:18 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:46:38 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:46:58 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:47:18 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:47:38 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
> Thu Jul 09 19:47:59 PDT 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6,
> java.net.UnknownHostException: unknown host: prod-hadoop-data02
>
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:129)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:679)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:360)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:358)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> Caused by: java.net.UnknownHostException: unknown host: prod-hadoop-data02
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(RpcClient.java:385)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
> at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1530)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.bulkLoadHFile(ClientProtos.java:30340)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1605)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1585)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:634)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:623)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
> ... 8 more
>
>
> No doubt, the exception clearly states that its not able to reach to the
> mentioned host, but unfortunately the host-ip entry was missing in the
> hosts file which I made sure to include in the hosts file.
>
> Moreover yesterday after including the host entry I re-ran the job(still
> it wasn't working with resume) and it worked as expected but today again
> its falling at the same point.
>
> Any helpful advice would be appreciated.
>
> Thanks,
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Vineet Mishra <cl...@gmail.com>.
Hi Luke,
We are using Amazon EC2 services.
Thanks,
On Mon, Jul 13, 2015 at 8:36 AM, Luke Han <lu...@gmail.com> wrote:
> You are running on AWS, EC2 or EMR? Which HBase version you are using?
>
> Thanks.
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> On Mon, Jul 13, 2015 at 10:50 AM, Sarnath <st...@gmail.com> wrote:
>
> > By hbase Cluster I meant the same one you are referring to. If your clus
> is
> > stable then it can either be ec2 which possibly is imposing limits on nwk
> > traffic, or some software version problems. Hope you are using supported
> > version of hbase and a stable version of kylin. I can't think of anything
> > more. Good luck.
> >
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Luke Han <lu...@gmail.com>.
You are running on AWS, EC2 or EMR? Which HBase version you are using?
Thanks.
Best Regards!
---------------------
Luke Han
On Mon, Jul 13, 2015 at 10:50 AM, Sarnath <st...@gmail.com> wrote:
> By hbase Cluster I meant the same one you are referring to. If your clus is
> stable then it can either be ec2 which possibly is imposing limits on nwk
> traffic, or some software version problems. Hope you are using supported
> version of hbase and a stable version of kylin. I can't think of anything
> more. Good luck.
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Sarnath <st...@gmail.com>.
By hbase Cluster I meant the same one you are referring to. If your clus is
stable then it can either be ec2 which possibly is imposing limits on nwk
traffic, or some software version problems. Hope you are using supported
version of hbase and a stable version of kylin. I can't think of anything
more. Good luck.
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Vineet Mishra <cl...@gmail.com>.
Sarnath,
What do u mean by hbase cluster? I don't have some separate cluster for
hadoop and hbase, hbase services are running on the same cluster which host
the hadoop and running other jobs and for no doubt the node is accessible
with healthy status.
On Sun, Jul 12, 2015 at 7:12 PM, Sarnath <st...@gmail.com> wrote:
> Is the problematic hostname reachable from all nodes of the hbase Cluster?
> If cloudera, check hbase Cluster health from cloudera manager.
>
Re: Kylin Cube Creation Fails at Load HFile to Hbase Table
Posted by Sarnath <st...@gmail.com>.
Is the problematic hostname reachable from all nodes of the hbase Cluster?
If cloudera, check hbase Cluster health from cloudera manager.