You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Randy Fox <rf...@connexity.com> on 2016/10/20 15:26:28 UTC
Parallelizing TableSnapshotScanner
Hi,
I have attempted to parallelize the TableSnapshotScanner by taking the ClientSideRegionScanner’s it generates and sending them each to their own thread to read the results. I am finding that I do not get all the rows per region when I do this. It seems the more parallelism the less data I get back. Any ideas on why these scanners cannot be accessed in parallel?
Thanks in advance,
Randy
Re: Parallelizing TableSnapshotScanner
Posted by Randy Fox <rf...@connexity.com>.
I can work on that. Since it is reading the restore data from a snapshot it is definitely not changing. I wanted to make sure there was not some inherent thread model under the scanner that prohibited this from working.
On 10/20/16, 8:30 AM, "Ted Yu" <yu...@gmail.com> wrote:
>Can you come up with unit test that shows what you observed ?
>You may have seen the javadoc for ClientSideRegionScanner :
>
> * A client scanner for a region opened for read-only on the client side.
>Assumes region data
>
> * is not changing.
>
>Cheers
>
>On Thu, Oct 20, 2016 at 8:26 AM, Randy Fox <rf...@connexity.com> wrote:
>
>> Hi,
>>
>> I have attempted to parallelize the TableSnapshotScanner by taking the
>> ClientSideRegionScanner’s it generates and sending them each to their own
>> thread to read the results. I am finding that I do not get all the rows
>> per region when I do this. It seems the more parallelism the less data I
>> get back. Any ideas on why these scanners cannot be accessed in parallel?
>>
>> Thanks in advance,
>>
>> Randy
>>
Re: Parallelizing TableSnapshotScanner
Posted by Ted Yu <yu...@gmail.com>.
Can you come up with unit test that shows what you observed ?
You may have seen the javadoc for ClientSideRegionScanner :
* A client scanner for a region opened for read-only on the client side.
Assumes region data
* is not changing.
Cheers
On Thu, Oct 20, 2016 at 8:26 AM, Randy Fox <rf...@connexity.com> wrote:
> Hi,
>
> I have attempted to parallelize the TableSnapshotScanner by taking the
> ClientSideRegionScanner’s it generates and sending them each to their own
> thread to read the results. I am finding that I do not get all the rows
> per region when I do this. It seems the more parallelism the less data I
> get back. Any ideas on why these scanners cannot be accessed in parallel?
>
> Thanks in advance,
>
> Randy
>