You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jack Chan <cd...@gmail.com> on 2013/10/23 06:07:30 UTC
How can I insert large image or video into HBase?
Hi All:
This could be a stupid question.But here it goes....
We knew that we can use "put" to insert some small files by converting it to bytes first.
But for a large file,I think we would better stream it first.
So,how can we insert the large file into HBase through Java code using the stream way?
Thanks and regards
Jack Chan.
Re: How can I insert large image or video into HBase?
Posted by Wei Tan <wt...@us.ibm.com>.
Roman, thanks for sharing your experience. Is your approach is somewhat
similar to Facebook's image store, Haystack?
I am very interested in knowing your use case, and what you actually mean
by class abstraction, internal write buffer, etc.
Best regards,
Wei
From: Roman Nikitchenko <ro...@nikitchenko.dp.ua>
To: user@hbase.apache.org,
Date: 10/23/2013 02:51 AM
Subject: Re: How can I insert large image or video into HBase?
We had need to store LOT of files (mostly not so big but they could be up
to few G) and we have decided to do it based on HBase. We store files in
column blocks.
Shortly solution is:
1. 2 Column families: single for metadata and single for content.
2. Class abstraction that provides client with stream to write file or
read
it.
3. Internal write buffer and internal buffer of formed puts so write speed
is really good. Up to 2 times better than HDFS on files below 128K.
4. If client uses buffered writes, place buffers 1:1 to columns
(segmentation control).
5. Seek is implemented based on 2 client filters (to limit column range
and
to get only qualifiers). So based on skip() we check what block shall we
buffer and set current position.
Advantages of this solution are:
- Small files problem is solved (shall I comment something here?).
- Thread safe without headaches.
- Possibility to use compression transparently.
- Metadata is really flexible (ok, HDFS can get it but again using HBase,
otherwise - small files problem is yours).
- Locality control due to regions (not possible with HDFS).
- Very effective MR processing due to previous point.
- Ability to use 'lightweight MR'
Disadvantages:
- Somewhat more complex client. We just have encapsulated this and don't
care any more. Right today I plan to add SHA1 hashes support.
- On large files (10M and more) we are notable slower than HDFS. Probably
it can be improved with MemStores configuration and so on but I just don't
care, for our needs it is enough.
These ideas should be enough to understand approach.
BTW I consider to publish this solution on GitHub to get some kind of
'community review'.
Best regards,
Roman.
On 23 October 2013 07:12, Jean-Marc Spaggiari
<je...@spaggiari.org>wrote:
> Put your file into HDFS and store only the name into HBase. HBase is not
> done do store large files.
>
> JM
>
>
> 2013/10/23 Jack Chan <cd...@gmail.com>
>
> > Hi All:
> >
> > This could be a stupid question.But here it goes....
> > We knew that we can use "put" to insert some small files by converting
it
> > to bytes first.
> > But for a large file,I think we would better stream it first.
> > So,how can we insert the large file into HBase through Java code using
> the
> > stream way?
> >
> > Thanks and regards
> >
> >
> >
> > Jack Chan.
>
Re: 答复: How can I insert large image or video into HBase?
Posted by Roman Nikitchenko <ro...@nikitchenko.dp.ua>.
As file is broken by blocks during any operation I think GC is loaded not
so much more than during plain HDFS operations (actually I thing less).
More serious load here is on MemStore and WAL but reliability is what I
want and yes, nothing is for free.
Best regards,
Roman.
On 23 October 2013 09:56, 谢良 <xi...@xiaomi.com> wrote:
> Do you care about low latency? if so, then maybe it's not a good choice
> to store big file into hbase, especially few G size, that's definitely will
> bring a GC hurt:)
>
> Best,
> Liang
>
>
答复: How can I insert large image or video into HBase?
Posted by 谢良 <xi...@xiaomi.com>.
Do you care about low latency? if so, then maybe it's not a good choice to store big file into hbase, especially few G size, that's definitely will bring a GC hurt:)
Best,
Liang
________________________________________
发件人: Roman Nikitchenko [roman@nikitchenko.dp.ua]
发送时间: 2013年10月23日 14:50
收件人: user@hbase.apache.org
主题: Re: How can I insert large image or video into HBase?
We had need to store LOT of files (mostly not so big but they could be up
to few G) and we have decided to do it based on HBase. We store files in
column blocks.
Shortly solution is:
1. 2 Column families: single for metadata and single for content.
2. Class abstraction that provides client with stream to write file or read
it.
3. Internal write buffer and internal buffer of formed puts so write speed
is really good. Up to 2 times better than HDFS on files below 128K.
4. If client uses buffered writes, place buffers 1:1 to columns
(segmentation control).
5. Seek is implemented based on 2 client filters (to limit column range and
to get only qualifiers). So based on skip() we check what block shall we
buffer and set current position.
Advantages of this solution are:
- Small files problem is solved (shall I comment something here?).
- Thread safe without headaches.
- Possibility to use compression transparently.
- Metadata is really flexible (ok, HDFS can get it but again using HBase,
otherwise - small files problem is yours).
- Locality control due to regions (not possible with HDFS).
- Very effective MR processing due to previous point.
- Ability to use 'lightweight MR'
Disadvantages:
- Somewhat more complex client. We just have encapsulated this and don't
care any more. Right today I plan to add SHA1 hashes support.
- On large files (10M and more) we are notable slower than HDFS. Probably
it can be improved with MemStores configuration and so on but I just don't
care, for our needs it is enough.
These ideas should be enough to understand approach.
BTW I consider to publish this solution on GitHub to get some kind of
'community review'.
Best regards,
Roman.
On 23 October 2013 07:12, Jean-Marc Spaggiari <je...@spaggiari.org>wrote:
> Put your file into HDFS and store only the name into HBase. HBase is not
> done do store large files.
>
> JM
>
>
> 2013/10/23 Jack Chan <cd...@gmail.com>
>
> > Hi All:
> >
> > This could be a stupid question.But here it goes....
> > We knew that we can use "put" to insert some small files by converting it
> > to bytes first.
> > But for a large file,I think we would better stream it first.
> > So,how can we insert the large file into HBase through Java code using
> the
> > stream way?
> >
> > Thanks and regards
> >
> >
> >
> > Jack Chan.
>
Re: How can I insert large image or video into HBase?
Posted by Roman Nikitchenko <ro...@nikitchenko.dp.ua>.
We had need to store LOT of files (mostly not so big but they could be up
to few G) and we have decided to do it based on HBase. We store files in
column blocks.
Shortly solution is:
1. 2 Column families: single for metadata and single for content.
2. Class abstraction that provides client with stream to write file or read
it.
3. Internal write buffer and internal buffer of formed puts so write speed
is really good. Up to 2 times better than HDFS on files below 128K.
4. If client uses buffered writes, place buffers 1:1 to columns
(segmentation control).
5. Seek is implemented based on 2 client filters (to limit column range and
to get only qualifiers). So based on skip() we check what block shall we
buffer and set current position.
Advantages of this solution are:
- Small files problem is solved (shall I comment something here?).
- Thread safe without headaches.
- Possibility to use compression transparently.
- Metadata is really flexible (ok, HDFS can get it but again using HBase,
otherwise - small files problem is yours).
- Locality control due to regions (not possible with HDFS).
- Very effective MR processing due to previous point.
- Ability to use 'lightweight MR'
Disadvantages:
- Somewhat more complex client. We just have encapsulated this and don't
care any more. Right today I plan to add SHA1 hashes support.
- On large files (10M and more) we are notable slower than HDFS. Probably
it can be improved with MemStores configuration and so on but I just don't
care, for our needs it is enough.
These ideas should be enough to understand approach.
BTW I consider to publish this solution on GitHub to get some kind of
'community review'.
Best regards,
Roman.
On 23 October 2013 07:12, Jean-Marc Spaggiari <je...@spaggiari.org>wrote:
> Put your file into HDFS and store only the name into HBase. HBase is not
> done do store large files.
>
> JM
>
>
> 2013/10/23 Jack Chan <cd...@gmail.com>
>
> > Hi All:
> >
> > This could be a stupid question.But here it goes....
> > We knew that we can use "put" to insert some small files by converting it
> > to bytes first.
> > But for a large file,I think we would better stream it first.
> > So,how can we insert the large file into HBase through Java code using
> the
> > stream way?
> >
> > Thanks and regards
> >
> >
> >
> > Jack Chan.
>
Re: Re: How can I insert large image or video into HBase?
Posted by Jack Chan <cd...@gmail.com>.
Big thanks to everybody :)
Sorry that I am not good at English communication.But It's really good that I can receive so many kind reply here.
Thanks to Michael for your kindly reply and recommending me Lars George and Nick Dimiduk's book,I think that is <HBase, The Definitive Guide> and <HBase in Action>,
Roman,I really hoping for your solution on GItHub.
JM,thanks for your encouragement :)
to 谢良: 多谢回复，呵呵
Best Regards.
Jack Chan.
From: Jean-Marc Spaggiari
Date: 2013-10-23 22:46
To: Michael Segel
CC: user; cdj0579
Subject: Re: How can I insert large image or video into HBase?
To reply to Jack: There is no stupid question. What is stupid is to go in the wrong direction, close its eyes, and run. Asking is always good.
You got a very nicely detailed respons from Michael, so I don't think I have anything to add ;)
JM
2013/10/23 Michael Segel <mi...@hotmail.com>
Actually you didn't.
Its actually a good question and JM gave you the short answer. ;-)
First to clarify... depending on the size of the image or video you may want to actually store the image/vid in a sequence file.
Then use HBase to index the file. (Its pretty much the same idea as what JM was saying but that you have to remember that even Vid and image files may be too large to fit into HBase yet too small to be separate files. )
This is one of the patterns that we've talked about at conferences. (Big Data TechCon) and its covered in Boris Lublinsky, Kevin Smith and Alexey Y's new book.
(Its a Wiley publication.)
The longer answer...
In HBase your row cannot exceed the width of the region. So you end up with very large regions, with a single row and you have to cap the size of the video or image.
HBase doesn't really perform well and of course performance would suck when you tried to insert or read the row.
(You can go through examples in either Lars George's book or Nick Dimiduk ?sp? book on what happens during reads and writes.) Unless the row is in cache, reads and writes to HBase are slower than reading or writing to HDFS.
The reason I like the question is that it forces people to think about how HBase works and why it may not be the best option as a stand alone solution. ;-)
On Oct 23, 2013, at 1:38 AM, Jack Chan <cd...@gmail.com> wrote:
> Hi JM,
>
> Thanks for your reply,It seems I really asked a stupid question :)
>
>
>
>
> Jack Chan.
>
> From: Jean-Marc Spaggiari
> Date: 2013-10-23 12:12
> To: user; cdj0579
> Subject: Re: How can I insert large image or video into HBase?
> Put your file into HDFS and store only the name into HBase. HBase is not done do store large files.
>
>
> JM
>
>
>
>
> 2013/10/23 Jack Chan <cd...@gmail.com>
>
> Hi All:
>
> This could be a stupid question.But here it goes....
> We knew that we can use "put" to insert some small files by converting it to bytes first.
> But for a large file,I think we would better stream it first.
> So,how can we insert the large file into HBase through Java code using the stream way?
>
> Thanks and regards
>
>
>
> Jack Chan.
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: How can I insert large image or video into HBase?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
To reply to Jack: There is no stupid question. What is stupid is to go in
the wrong direction, close its eyes, and run. Asking is always good.
You got a very nicely detailed respons from Michael, so I don't think I
have anything to add ;)
JM
2013/10/23 Michael Segel <mi...@hotmail.com>
> Actually you didn't.
>
> Its actually a good question and JM gave you the short answer. ;-)
>
> First to clarify... depending on the size of the image or video you may
> want to actually store the image/vid in a sequence file.
> Then use HBase to index the file. (Its pretty much the same idea as what
> JM was saying but that you have to remember that even Vid and image files
> may be too large to fit into HBase yet too small to be separate files. )
>
> This is one of the patterns that we've talked about at conferences. (Big
> Data TechCon) and its covered in Boris Lublinsky, Kevin Smith and Alexey
> Y's new book.
> (Its a Wiley publication.)
>
> The longer answer...
>
> In HBase your row cannot exceed the width of the region. So you end up
> with very large regions, with a single row and you have to cap the size of
> the video or image.
> HBase doesn't really perform well and of course performance would suck
> when you tried to insert or read the row.
> (You can go through examples in either Lars George's book or Nick Dimiduk
> ?sp? book on what happens during reads and writes.) Unless the row is in
> cache, reads and writes to HBase are slower than reading or writing to HDFS.
>
> The reason I like the question is that it forces people to think about how
> HBase works and why it may not be the best option as a stand alone
> solution. ;-)
>
> On Oct 23, 2013, at 1:38 AM, Jack Chan <cd...@gmail.com> wrote:
>
> > Hi JM,
> >
> > Thanks for your reply,It seems I really asked a stupid question :)
> >
> >
> >
> >
> > Jack Chan.
> >
> > From: Jean-Marc Spaggiari
> > Date: 2013-10-23 12:12
> > To: user; cdj0579
> > Subject: Re: How can I insert large image or video into HBase?
> > Put your file into HDFS and store only the name into HBase. HBase is not
> done do store large files.
> >
> >
> > JM
> >
> >
> >
> >
> > 2013/10/23 Jack Chan <cd...@gmail.com>
> >
> > Hi All:
> >
> > This could be a stupid question.But here it goes....
> > We knew that we can use "put" to insert some small files by converting
> it to bytes first.
> > But for a large file,I think we would better stream it first.
> > So,how can we insert the large file into HBase through Java code using
> the stream way?
> >
> > Thanks and regards
> >
> >
> >
> > Jack Chan.
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: How can I insert large image or video into HBase?
Posted by Michael Segel <mi...@hotmail.com>.
Actually you didn't.
Its actually a good question and JM gave you the short answer. ;-)
First to clarify... depending on the size of the image or video you may want to actually store the image/vid in a sequence file.
Then use HBase to index the file. (Its pretty much the same idea as what JM was saying but that you have to remember that even Vid and image files may be too large to fit into HBase yet too small to be separate files. )
This is one of the patterns that we've talked about at conferences. (Big Data TechCon) and its covered in Boris Lublinsky, Kevin Smith and Alexey Y's new book.
(Its a Wiley publication.)
The longer answer...
In HBase your row cannot exceed the width of the region. So you end up with very large regions, with a single row and you have to cap the size of the video or image.
HBase doesn't really perform well and of course performance would suck when you tried to insert or read the row.
(You can go through examples in either Lars George's book or Nick Dimiduk ?sp? book on what happens during reads and writes.) Unless the row is in cache, reads and writes to HBase are slower than reading or writing to HDFS.
The reason I like the question is that it forces people to think about how HBase works and why it may not be the best option as a stand alone solution. ;-)
On Oct 23, 2013, at 1:38 AM, Jack Chan <cd...@gmail.com> wrote:
> Hi JM,
>
> Thanks for your reply,It seems I really asked a stupid question :)
>
>
>
>
> Jack Chan.
>
> From: Jean-Marc Spaggiari
> Date: 2013-10-23 12:12
> To: user; cdj0579
> Subject: Re: How can I insert large image or video into HBase?
> Put your file into HDFS and store only the name into HBase. HBase is not done do store large files.
>
>
> JM
>
>
>
>
> 2013/10/23 Jack Chan <cd...@gmail.com>
>
> Hi All:
>
> This could be a stupid question.But here it goes....
> We knew that we can use "put" to insert some small files by converting it to bytes first.
> But for a large file,I think we would better stream it first.
> So,how can we insert the large file into HBase through Java code using the stream way?
>
> Thanks and regards
>
>
>
> Jack Chan.
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Re: How can I insert large image or video into HBase?
Posted by Jack Chan <cd...@gmail.com>.
Hi JM,
Thanks for your reply,It seems I really asked a stupid question :)
Jack Chan.
From: Jean-Marc Spaggiari
Date: 2013-10-23 12:12
To: user; cdj0579
Subject: Re: How can I insert large image or video into HBase?
Put your file into HDFS and store only the name into HBase. HBase is not done do store large files.
JM
2013/10/23 Jack Chan <cd...@gmail.com>
Hi All:
This could be a stupid question.But here it goes....
We knew that we can use "put" to insert some small files by converting it to bytes first.
But for a large file,I think we would better stream it first.
So,how can we insert the large file into HBase through Java code using the stream way?
Thanks and regards
Jack Chan.
Re: How can I insert large image or video into HBase?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Put your file into HDFS and store only the name into HBase. HBase is not
done do store large files.
JM
2013/10/23 Jack Chan <cd...@gmail.com>
> Hi All:
>
> This could be a stupid question.But here it goes....
> We knew that we can use "put" to insert some small files by converting it
> to bytes first.
> But for a large file,I think we would better stream it first.
> So,how can we insert the large file into HBase through Java code using the
> stream way?
>
> Thanks and regards
>
>
>
> Jack Chan.