You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Liu, Raymond" <ra...@intel.com> on 2013/01/17 07:44:03 UTC

How is DataXceiver been used?

Hi

	I have a table with about 24 region on the one regionserver, and each region have about 20 block files on hdfs.
	The xceiverCount is set to 1024, I have thought that this is quite enough since at most 480 blocks will be opened.
	While when I do a MR job to scan the table, with 24 map task each open and scan a different region at the same time, it turn out that the DataXceiver is run out...

	I am a little bit puzzled, those blocks will only be read by one task, then shouldn't region server scan blocks one by one? And since there are 480 blocks at most, how can it use up dataXceiver?

Best Regards,
Raymond Liu

RE: How is DataXceiver been used?

Posted by "Liu, Raymond" <ra...@intel.com>.
Hi Tariq

Thanks, this blog is very informational!

I roughly figure out the usage of xceiver. While the weird thing is even according to this blog, I should have far enough xceiever in my full table scan job. Especially if it count the number of storefile instead of real blk_xxxx tfile on nativeFS. Then I only have around 24 storefiles for 24 region since I do major compact them. And then it exceeding 1024 limit during full table scan.... 

Might need more investigation into my case to check out what wrong with it.


> 
> Hello Raymond,
> 
>             You might find this
> link<http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/>helpful.
> It explains the problem
> and solution in great detail.
> 
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
> 
> 
> On Thu, Jan 17, 2013 at 12:14 PM, Liu, Raymond
> <ra...@intel.com>wrote:
> 
> > Hi
> >
> >         I have a table with about 24 region on the one regionserver,
> > and each region have about 20 block files on hdfs.
> >         The xceiverCount is set to 1024, I have thought that this is
> > quite enough since at most 480 blocks will be opened.
> >         While when I do a MR job to scan the table, with 24 map task
> > each open and scan a different region at the same time, it turn out
> > that the DataXceiver is run out...
> >
> >         I am a little bit puzzled, those blocks will only be read by
> > one task, then shouldn't region server scan blocks one by one? And
> > since there are 480 blocks at most, how can it use up dataXceiver?
> >
> > Best Regards,
> > Raymond Liu
> >
> >

Re: How is DataXceiver been used?

Posted by Mohammad Tariq <do...@gmail.com>.
Hello Raymond,

            You might find this
link<http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/>helpful.
It explains the problem
and solution in great detail.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Thu, Jan 17, 2013 at 12:14 PM, Liu, Raymond <ra...@intel.com>wrote:

> Hi
>
>         I have a table with about 24 region on the one regionserver, and
> each region have about 20 block files on hdfs.
>         The xceiverCount is set to 1024, I have thought that this is quite
> enough since at most 480 blocks will be opened.
>         While when I do a MR job to scan the table, with 24 map task each
> open and scan a different region at the same time, it turn out that the
> DataXceiver is run out...
>
>         I am a little bit puzzled, those blocks will only be read by one
> task, then shouldn't region server scan blocks one by one? And since there
> are 480 blocks at most, how can it use up dataXceiver?
>
> Best Regards,
> Raymond Liu
>
>