You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Randy Fox <rf...@connexity.com> on 2016/10/20 15:26:28 UTC

Parallelizing TableSnapshotScanner

Hi,

I have attempted to parallelize the TableSnapshotScanner by taking the ClientSideRegionScanner’s it generates and sending them each to their own thread to read the results.  I am finding that I do not get all the rows per region when I do this.  It seems the more parallelism the less data I get back.  Any ideas on why these scanners cannot be accessed in parallel?

Thanks in advance,

Randy

Re: Parallelizing TableSnapshotScanner

Posted by Randy Fox <rf...@connexity.com>.
I can work on that.  Since it is reading the restore data from a snapshot it is definitely not changing.  I wanted to make sure there was not some inherent thread model under the scanner that prohibited this from working.




On 10/20/16, 8:30 AM, "Ted Yu" <yu...@gmail.com> wrote:

>Can you come up with unit test that shows what you observed ?
>You may have seen the javadoc for ClientSideRegionScanner :
>
> * A client scanner for a region opened for read-only on the client side.
>Assumes region data
>
> * is not changing.
>
>Cheers
>
>On Thu, Oct 20, 2016 at 8:26 AM, Randy Fox <rf...@connexity.com> wrote:
>
>> Hi,
>>
>> I have attempted to parallelize the TableSnapshotScanner by taking the
>> ClientSideRegionScanner’s it generates and sending them each to their own
>> thread to read the results.  I am finding that I do not get all the rows
>> per region when I do this.  It seems the more parallelism the less data I
>> get back.  Any ideas on why these scanners cannot be accessed in parallel?
>>
>> Thanks in advance,
>>
>> Randy
>>

Re: Parallelizing TableSnapshotScanner

Posted by Ted Yu <yu...@gmail.com>.
Can you come up with unit test that shows what you observed ?
You may have seen the javadoc for ClientSideRegionScanner :

 * A client scanner for a region opened for read-only on the client side.
Assumes region data

 * is not changing.

Cheers

On Thu, Oct 20, 2016 at 8:26 AM, Randy Fox <rf...@connexity.com> wrote:

> Hi,
>
> I have attempted to parallelize the TableSnapshotScanner by taking the
> ClientSideRegionScanner’s it generates and sending them each to their own
> thread to read the results.  I am finding that I do not get all the rows
> per region when I do this.  It seems the more parallelism the less data I
> get back.  Any ideas on why these scanners cannot be accessed in parallel?
>
> Thanks in advance,
>
> Randy
>