You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Cheng Su <sc...@gmail.com> on 2012/11/07 04:07:37 UTC
Hbase connection closed when query multiple complicated hql with
hive+hbase integration
Hi, all.
I have a hive + hbase integration cluster. I met a performance issue.
I executed some complicated hql, but only the first one is actually running.
The rests are showed "running" on the job track web ui, but task
attempt failed with exceptions below:
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
closed
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
closed
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
... 9 more
So actually the rests are queuing.
I guess it might be because the count(?) of hbase handler is too
small, which handle access request from hive.
So that when a big job occupied all handles the rests have to wait
until the handlers are released.
Is my assumption right? and what settings should I tuning?
Thanks.
--
Regards,
Cheng Su
Re: Hbase connection closed when query multiple complicated hql with
hive+hbase integration
Posted by Cheng Su <sc...@gmail.com>.
The exceptions seem to be another problem.
They all happened on one node.
And after the task attempts failed at that node,
retried on other nodes and no exceptions.
So that, the exception maybe have nothing to do with the performance issue.
On Wed, Nov 7, 2012 at 11:07 AM, Cheng Su <sc...@gmail.com> wrote:
> Hi, all.
>
> I have a hive + hbase integration cluster. I met a performance issue.
> I executed some complicated hql, but only the first one is actually running.
> The rests are showed "running" on the job track web ui, but task
> attempt failed with exceptions below:
>
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
> closed
> at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
> closed
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
> at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
> ... 9 more
>
> So actually the rests are queuing.
>
> I guess it might be because the count(?) of hbase handler is too
> small, which handle access request from hive.
> So that when a big job occupied all handles the rests have to wait
> until the handlers are released.
>
> Is my assumption right? and what settings should I tuning?
>
> Thanks.
>
> --
>
> Regards,
> Cheng Su
--
Regards,
Cheng Su
Re: Hbase connection closed when query multiple complicated hql with
hive+hbase integration
Posted by Cheng Su <sc...@gmail.com>.
The exceptions seem to be another problem.
They all happened on one node.
And after the task attempts failed at that node,
retried on other nodes and no exceptions.
So that, the exception maybe have nothing to do with the performance issue.
On Wed, Nov 7, 2012 at 11:07 AM, Cheng Su <sc...@gmail.com> wrote:
> Hi, all.
>
> I have a hive + hbase integration cluster. I met a performance issue.
> I executed some complicated hql, but only the first one is actually running.
> The rests are showed "running" on the job track web ui, but task
> attempt failed with exceptions below:
>
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
> closed
> at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
> closed
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
> at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
> ... 9 more
>
> So actually the rests are queuing.
>
> I guess it might be because the count(?) of hbase handler is too
> small, which handle access request from hive.
> So that when a big job occupied all handles the rests have to wait
> until the handlers are released.
>
> Is my assumption right? and what settings should I tuning?
>
> Thanks.
>
> --
>
> Regards,
> Cheng Su
--
Regards,
Cheng Su