You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Snake <yf...@xingcloud.com> on 2013/03/21 10:05:34 UTC
Is there a way to only scan data in memstore
Hi,
I just want to scan data in memstore, but I saw InternalScan is private class. Is there a way i can do it without change the hbase source code?
Thanks,
Snake
Re: Is there a way to only scan data in memstore
Posted by Enis Söztutar <en...@gmail.com>.
I would caution against going that route for various reasons:
- Correctness : You can never be sure to sync the memstore flushes and
compactions changing the files under you.
- Security : All files from HBase are owned by HBase user. Other
users should not be able to read it.
- Compatibility : HBase gives zero guarantees about how the data is stored
internally. At any point, the layout, file format, block encodings, etc
might change.
Enis
On Thu, Mar 21, 2013 at 3:34 AM, Snake <yf...@xingcloud.com> wrote:
> We are reading hfile directly by knowing the HDFS path. Thanks guys.
> We'll hack some codes to achieve our goal.
> 在 2013-3-21，下午5:56， ramkrishna vasudevan 写道：
>
> > Memstore
>
>
Re: Is there a way to only scan data in memstore
Posted by Snake <yf...@xingcloud.com>.
We are reading hfile directly by knowing the HDFS path. Thanks guys. We'll hack some codes to achieve our goal.
在 2013-3-21，下午5:56， ramkrishna vasudevan 写道：
> Memstore
Re: Is there a way to only scan data in memstore
Posted by ramkrishna vasudevan <ra...@gmail.com>.
The way that Anoop has suggested will make you issue a scan command from
the client but the CP hook will tel you what are the KVs in the memstore
based on that particular scan's current read point.
I think to scan the HFiles directly are you using MapReduce? Or you are
directly reading the HFiles knowing the HDFS path?
Currently i don think there is any way to access a Memstore directlly,
leave even the Memstore you cannot access the HRegion directly if am not
wrong.
Regards
Ram
On Thu, Mar 21, 2013 at 3:14 PM, Snake <yf...@xingcloud.com> wrote:
> We are attempting to implement hbase as data source in drill. We don't
> want to use hbase client to scan data(performance consideration). We
> divided the scan process into two parts, read hfile directly and read data
> in memstore from region server.
> 在 2013-3-21，下午5:31， Anoop Sam John 写道：
>
> > How you can be sure abt data will be in memstore only. What if in btw
> flush happening? Which version in use?
> > In 94.x version (I am not sure abt the .x version no#) there is
> preStoreScannerOpen() CP hook. This impl can return a KVScanner for a store
> (In your impl the scanner can be only for Memstore?). Pls see how this hook
> is being used and how the StoreScanner is being created.
> >
> > Can you think of control this using TimeRange on Scan ?
> >
> > -Anoop-
> > ________________________________________
> > From: Snake [yfwang@xingcloud.com]
> > Sent: Thursday, March 21, 2013 2:35 PM
> > To: user@hbase.apache.org
> > Subject: Is there a way to only scan data in memstore
> >
> > Hi,
> >
> > I just want to scan data in memstore, but I saw InternalScan is private
> class. Is there a way i can do it without change the hbase source code?
> >
> > Thanks,
> > Snake
>
>
Re: Is there a way to only scan data in memstore
Posted by Snake <yf...@xingcloud.com>.
Thanks Lars. That issue is very similar with our requirement. We'll keep focus on it and may contribute some code on it.
Thanks,
Snake
在 2013-3-23，下午12:00， lars hofhansl 写道：
> You might find HBASE-8185 (https://issues.apache.org/jira/browse/HBASE-8185) interesting.
>
> -- Lars
>
>
>
> ________________________________
> From: Snake <yf...@xingcloud.com>
> To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Friday, March 22, 2013 12:47 AM
> Subject: Re: Is there a way to only scan data in memstore
>
> Our usage scenario is that when the hbase client and hfile are in the same machine(read local), we don't want to hbase client to communicate with regionserver and then send back the data. Serialization, deserialization and network transmission are time-consuming. The best way is hbase client can read local data directly. It's great if hbase client can consider these kinds of thing to improve scan performance.
> 在 2013-3-22，下午12:19， lars hofhansl 写道：
>
>> provide
Re: Is there a way to only scan data in memstore
Posted by lars hofhansl <la...@apache.org>.
You might find HBASE-8185 (https://issues.apache.org/jira/browse/HBASE-8185) interesting.
-- Lars
________________________________
From: Snake <yf...@xingcloud.com>
To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Friday, March 22, 2013 12:47 AM
Subject: Re: Is there a way to only scan data in memstore
Our usage scenario is that when the hbase client and hfile are in the same machine(read local), we don't want to hbase client to communicate with regionserver and then send back the data. Serialization, deserialization and network transmission are time-consuming. The best way is hbase client can read local data directly. It's great if hbase client can consider these kinds of thing to improve scan performance.
在 2013-3-22，下午12:19， lars hofhansl 写道：
> provide
Re: Is there a way to only scan data in memstore
Posted by Snake <yf...@xingcloud.com>.
Our usage scenario is that when the hbase client and hfile are in the same machine(read local), we don't want to hbase client to communicate with regionserver and then send back the data. Serialization, deserialization and network transmission are time-consuming. The best way is hbase client can read local data directly. It's great if hbase client can consider these kinds of thing to improve scan performance.
在 2013-3-22，下午12:19， lars hofhansl 写道：
> provide
Re: Is there a way to only scan data in memstore
Posted by lars hofhansl <la...@apache.org>.
We should talk.
If you can envision an interface that is faster, maybe we can provide the right shortcut or change things around in the client to access the data at the right stage or format.
________________________________
From: Snake <yf...@xingcloud.com>
To: user@hbase.apache.org
Sent: Thursday, March 21, 2013 2:44 AM
Subject: Re: Is there a way to only scan data in memstore
We are attempting to implement hbase as data source in drill. We don't want to use hbase client to scan data(performance consideration). We divided the scan process into two parts, read hfile directly and read data in memstore from region server.
在 2013-3-21，下午5:31， Anoop Sam John 写道：
> How you can be sure abt data will be in memstore only. What if in btw flush happening? Which version in use?
> In 94.x version (I am not sure abt the .x version no#) there is preStoreScannerOpen() CP hook. This impl can return a KVScanner for a store (In your impl the scanner can be only for Memstore?). Pls see how this hook is being used and how the StoreScanner is being created.
>
> Can you think of control this using TimeRange on Scan ?
>
> -Anoop-
> ________________________________________
> From: Snake [yfwang@xingcloud.com]
> Sent: Thursday, March 21, 2013 2:35 PM
> To: user@hbase.apache.org
> Subject: Is there a way to only scan data in memstore
>
> Hi,
>
> I just want to scan data in memstore, but I saw InternalScan is private class. Is there a way i can do it without change the hbase source code?
>
> Thanks,
> Snake
Re: Is there a way to only scan data in memstore
Posted by Snake <yf...@xingcloud.com>.
We are attempting to implement hbase as data source in drill. We don't want to use hbase client to scan data(performance consideration). We divided the scan process into two parts, read hfile directly and read data in memstore from region server.
在 2013-3-21，下午5:31， Anoop Sam John 写道：
> How you can be sure abt data will be in memstore only. What if in btw flush happening? Which version in use?
> In 94.x version (I am not sure abt the .x version no#) there is preStoreScannerOpen() CP hook. This impl can return a KVScanner for a store (In your impl the scanner can be only for Memstore?). Pls see how this hook is being used and how the StoreScanner is being created.
>
> Can you think of control this using TimeRange on Scan ?
>
> -Anoop-
> ________________________________________
> From: Snake [yfwang@xingcloud.com]
> Sent: Thursday, March 21, 2013 2:35 PM
> To: user@hbase.apache.org
> Subject: Is there a way to only scan data in memstore
>
> Hi,
>
> I just want to scan data in memstore, but I saw InternalScan is private class. Is there a way i can do it without change the hbase source code?
>
> Thanks,
> Snake
RE: Is there a way to only scan data in memstore
Posted by Anoop Sam John <an...@huawei.com>.
How you can be sure abt data will be in memstore only. What if in btw flush happening? Which version in use?
In 94.x version (I am not sure abt the .x version no#) there is preStoreScannerOpen() CP hook. This impl can return a KVScanner for a store (In your impl the scanner can be only for Memstore?). Pls see how this hook is being used and how the StoreScanner is being created.
Can you think of control this using TimeRange on Scan ?
-Anoop-
________________________________________
From: Snake [yfwang@xingcloud.com]
Sent: Thursday, March 21, 2013 2:35 PM
To: user@hbase.apache.org
Subject: Is there a way to only scan data in memstore
Hi,
I just want to scan data in memstore, but I saw InternalScan is private class. Is there a way i can do it without change the hbase source code?
Thanks,
Snake