You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by xtliwen <xt...@gmail.com> on 2011/10/31 08:17:08 UTC
save video with hbase
Hi everybody,
When the client visit the video of a website through my website, it will be transcoded with our video codec server.As the time goes on,the quantities of the videos is quite large. Usuallly, a video can be tanscoded to serveral levels,so a original video is corresponding to multi transcoded videos . Now,we plan to record the video files with hbase. There are two problems with the hbase:
1 the video file is too large (100M avg.)
2 we require the transcoded video must can be read while it was been writing
So,can anybody give some suggestions, Thanks.
2011-10-31
regards
xtliwen
Re: save video with hbase
Posted by "Stanley C. Burnitt" <st...@globo.com>.
Mayuresh's suggestion is how I am dealing with this problem.
I save metadata about big files in an HBase row, including the path to the
Hadoop file containing the actual video -- or any other binary -- data.
I save large files (> 2GB) as single files in HDFS.
I save "medium sized" files (1MB > 2GB) as Hadoop TFile records. I
periodically merge smaller TFiles into 1 large TFile, and update the
corresponding hbase row to point to that new TFile.
Files <= 1MB are stored in the HBase table row.
Each hbase row holds
(a) the actual data -- for files <= 1MB
or (b) HDFS.File.Path, -- for files > 2GB
or (c) HDFS TFile.Path and TFile.Key -- for files between 1MB and 2GB in
size.
Bundling the 'medium sized files' (1MB - 2GB) in hadoop TFiles is my
solution to the 'small files problem'; it helps reduce the Hadoop
namenode's workload.
Regards,
Stan
2011/10/31 xtliwen <xt...@gmail.com>
> Hi everybody,
> When the client visit the video of a website through my website, it will
> be transcoded with our video codec server.As the time goes on,the
> quantities of the videos is quite large. Usuallly, a video can be tanscoded
> to serveral levels,so a original video is corresponding to multi transcoded
> videos . Now,we plan to record the video files with hbase. There are two
> problems with the hbase:
> 1 the video file is too large (100M avg.)
> 2 we require the transcoded video must can be read while it was been
> writing
>
> So,can anybody give some suggestions, Thanks.
>
> 2011-10-31
>
>
>
> regards
> xtliwen
>
Re: Re: save video with hbase
Posted by Mayuresh <ma...@gmail.com>.
Just curious.... Why not use hadoop directly for this?
On Nov 1, 2011 7:08 PM, "Joey Echeverria" <jo...@cloudera.com> wrote:
> I would save them in different rows. You could do that with a row key
> that looks something like this:
>
> <video name>\0<block number>
>
> You can use any separator there, not just \0, that isn't a valid video
> name or block number. This way you could skip to the block of the
> video that you need, but you're not forcing all of the blocks of a
> single video to be on one region server. Since regions are split on
> row boundaries, you run into some of the same issues with having a
> 100M+ row as having 100M in a single column.
>
> -Joey
>
> 2011/10/31 xtliwen <xt...@gmail.com>:
> > You are right,100M is too big ,so I plan to split the video to multi
> parts(avg 5M ), and save in different colums.
> >
> >
> > 2011-11-01
> >
> >
> >
> > xtliwen
> >
> >
> >
> > 发件人： Doug Meil
> > 发送时间： 2011-11-01 09:54:47
> > 收件人： user@hbase.apache.org
> > 抄送：
> > 主题： Re: save video with hbase
> >
> > Hi there, see...
> > http://hbase.apache.org/book.html#supported.datatypes
> > ... while you can store anything that can be converted to a byte-array in
> > HBase, as JD says 100meg is too big.
> > On 10/31/11 9:09 PM, "xtliwen" <xt...@gmail.com> wrote:
> >>
> >>I would like to save the data using hbase.The hbase is designed for small
> >>data,such as log.Maybe the video is too large for a
> >>row,so I am worried about the performance of the hbase in storing the
> >>video data.
> >>
> >>2011-11-01
> >>
> >>
> >>
> >>xtliwen
> >>
> >>
> >>
> >>发件人： Jean-Daniel Cryans
> >>发送时间： 2011-11-01 08:07:02
> >>收件人： user
> >>抄送：
> >>主题： Re: save video with hbase
> >>
> >>Why would you use HBase for that?
> >>Regarding your two questions:
> >>On Mon, Oct 31, 2011 at 4:17 PM, xtliwen <xt...@gmail.com> wrote:
> >>> Hi everybody,
> >>> When the client visit the video of a website through my website, it
> >>>will be transcoded with our video codec server.As the time goes on,the
> >>>quantities of the videos is quite large. Usuallly, a video can be
> >>>tanscoded to serveral levels,so a original video is corresponding to
> >>>multi transcoded videos . Now,we plan to record the video files with
> >>>hbase. There are two problems with the hbase:
> >>> 1 the video file is too large (100M avg.)
> >>Why do you say that? The client side limit by default is -1
> >>(disabled), and if there's any other limit they are just set
> >>arbitrarily.
> >>> 2 we require the transcoded video must can be read while it was been
> >>>writing
> >>That sounds like an application-level problem, not HBase.
> >>>
> >>> So,can anybody give some suggestions, Thanks.
> >>>
> >>> 2011-10-31
> >>>
> >>>
> >>>
> >>> regards
> >>> xtliwen
> >>>
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>
Re: Re: save video with hbase
Posted by Joey Echeverria <jo...@cloudera.com>.
I would save them in different rows. You could do that with a row key
that looks something like this:
<video name>\0<block number>
You can use any separator there, not just \0, that isn't a valid video
name or block number. This way you could skip to the block of the
video that you need, but you're not forcing all of the blocks of a
single video to be on one region server. Since regions are split on
row boundaries, you run into some of the same issues with having a
100M+ row as having 100M in a single column.
-Joey
2011/10/31 xtliwen <xt...@gmail.com>:
> You are right,100M is too big ,so I plan to split the video to multi parts(avg 5M ), and save in different colums.
>
>
> 2011-11-01
>
>
>
> xtliwen
>
>
>
> 发件人： Doug Meil
> 发送时间： 2011-11-01 09:54:47
> 收件人： user@hbase.apache.org
> 抄送：
> 主题： Re: save video with hbase
>
> Hi there, see...
> http://hbase.apache.org/book.html#supported.datatypes
> ... while you can store anything that can be converted to a byte-array in
> HBase, as JD says 100meg is too big.
> On 10/31/11 9:09 PM, "xtliwen" <xt...@gmail.com> wrote:
>>
>>I would like to save the data using hbase.The hbase is designed for small
>>data,such as log.Maybe the video is too large for a
>>row,so I am worried about the performance of the hbase in storing the
>>video data.
>>
>>2011-11-01
>>
>>
>>
>>xtliwen
>>
>>
>>
>>发件人： Jean-Daniel Cryans
>>发送时间： 2011-11-01 08:07:02
>>收件人： user
>>抄送：
>>主题： Re: save video with hbase
>>
>>Why would you use HBase for that?
>>Regarding your two questions:
>>On Mon, Oct 31, 2011 at 4:17 PM, xtliwen <xt...@gmail.com> wrote:
>>> Hi everybody,
>>> When the client visit the video of a website through my website, it
>>>will be transcoded with our video codec server.As the time goes on,the
>>>quantities of the videos is quite large. Usuallly, a video can be
>>>tanscoded to serveral levels,so a original video is corresponding to
>>>multi transcoded videos . Now,we plan to record the video files with
>>>hbase. There are two problems with the hbase:
>>> 1 the video file is too large (100M avg.)
>>Why do you say that? The client side limit by default is -1
>>(disabled), and if there's any other limit they are just set
>>arbitrarily.
>>> 2 we require the transcoded video must can be read while it was been
>>>writing
>>That sounds like an application-level problem, not HBase.
>>>
>>> So,can anybody give some suggestions, Thanks.
>>>
>>> 2011-10-31
>>>
>>>
>>>
>>> regards
>>> xtliwen
>>>
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: Re: save video with hbase
Posted by xtliwen <xt...@gmail.com>.
You are right,100M is too big ,so I plan to split the video to multi parts(avg 5M ), and save in different colums.
2011-11-01
xtliwen
发件人： Doug Meil
发送时间： 2011-11-01 09:54:47
收件人： user@hbase.apache.org
抄送：
主题： Re: save video with hbase
Hi there, see...
http://hbase.apache.org/book.html#supported.datatypes
... while you can store anything that can be converted to a byte-array in
HBase, as JD says 100meg is too big.
On 10/31/11 9:09 PM, "xtliwen" <xt...@gmail.com> wrote:
>
>I would like to save the data using hbase.The hbase is designed for small
>data,such as log.Maybe the video is too large for a
>row,so I am worried about the performance of the hbase in storing the
>video data.
>
>2011-11-01
>
>
>
>xtliwen
>
>
>
>发件人： Jean-Daniel Cryans
>发送时间： 2011-11-01 08:07:02
>收件人： user
>抄送：
>主题： Re: save video with hbase
>
>Why would you use HBase for that?
>Regarding your two questions:
>On Mon, Oct 31, 2011 at 4:17 PM, xtliwen <xt...@gmail.com> wrote:
>> Hi everybody,
>> When the client visit the video of a website through my website, it
>>will be transcoded with our video codec server.As the time goes on,the
>>quantities of the videos is quite large. Usuallly, a video can be
>>tanscoded to serveral levels,so a original video is corresponding to
>>multi transcoded videos . Now,we plan to record the video files with
>>hbase. There are two problems with the hbase:
>> 1 the video file is too large (100M avg.)
>Why do you say that? The client side limit by default is -1
>(disabled), and if there's any other limit they are just set
>arbitrarily.
>> 2 we require the transcoded video must can be read while it was been
>>writing
>That sounds like an application-level problem, not HBase.
>>
>> So,can anybody give some suggestions, Thanks.
>>
>> 2011-10-31
>>
>>
>>
>> regards
>> xtliwen
>>
Re: save video with hbase
Posted by Doug Meil <do...@explorysmedical.com>.
Hi there, see...
http://hbase.apache.org/book.html#supported.datatypes
... while you can store anything that can be converted to a byte-array in
HBase, as JD says 100meg is too big.
On 10/31/11 9:09 PM, "xtliwen" <xt...@gmail.com> wrote:
>
>I would like to save the data using hbase.The hbase is designed for small
>data,such as log.Maybe the video is too large for a
>row,so I am worried about the performance of the hbase in storing the
>video data.
>
>2011-11-01
>
>
>
>xtliwen
>
>
>
>发件人： Jean-Daniel Cryans
>发送时间： 2011-11-01 08:07:02
>收件人： user
>抄送：
>主题： Re: save video with hbase
>
>Why would you use HBase for that?
>Regarding your two questions:
>On Mon, Oct 31, 2011 at 4:17 PM, xtliwen <xt...@gmail.com> wrote:
>> Hi everybody,
>> When the client visit the video of a website through my website, it
>>will be transcoded with our video codec server.As the time goes on,the
>>quantities of the videos is quite large. Usuallly, a video can be
>>tanscoded to serveral levels,so a original video is corresponding to
>>multi transcoded videos . Now,we plan to record the video files with
>>hbase. There are two problems with the hbase:
>> 1 the video file is too large (100M avg.)
>Why do you say that? The client side limit by default is -1
>(disabled), and if there's any other limit they are just set
>arbitrarily.
>> 2 we require the transcoded video must can be read while it was been
>>writing
>That sounds like an application-level problem, not HBase.
>>
>> So,can anybody give some suggestions, Thanks.
>>
>> 2011-10-31
>>
>>
>>
>> regards
>> xtliwen
>>
Re: Re: save video with hbase
Posted by xtliwen <xt...@gmail.com>.
I would like to save the data using hbase.The hbase is designed for small data,such as log.Maybe the video is too large for a
row,so I am worried about the performance of the hbase in storing the video data.
2011-11-01
xtliwen
发件人： Jean-Daniel Cryans
发送时间： 2011-11-01 08:07:02
收件人： user
抄送：
主题： Re: save video with hbase
Why would you use HBase for that?
Regarding your two questions:
On Mon, Oct 31, 2011 at 4:17 PM, xtliwen <xt...@gmail.com> wrote:
> Hi everybody,
> When the client visit the video of a website through my website, it will be transcoded with our video codec server.As the time goes on,the quantities of the videos is quite large. Usuallly, a video can be tanscoded to serveral levels,so a original video is corresponding to multi transcoded videos . Now,we plan to record the video files with hbase. There are two problems with the hbase:
> 1 the video file is too large (100M avg.)
Why do you say that? The client side limit by default is -1
(disabled), and if there's any other limit they are just set
arbitrarily.
> 2 we require the transcoded video must can be read while it was been writing
That sounds like an application-level problem, not HBase.
>
> So,can anybody give some suggestions, Thanks.
>
> 2011-10-31
>
>
>
> regards
> xtliwen
>
Re: save video with hbase
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Why would you use HBase for that?
Regarding your two questions:
On Mon, Oct 31, 2011 at 4:17 PM, xtliwen <xt...@gmail.com> wrote:
> Hi everybody,
> When the client visit the video of a website through my website, it will be transcoded with our video codec server.As the time goes on,the quantities of the videos is quite large. Usuallly, a video can be tanscoded to serveral levels,so a original video is corresponding to multi transcoded videos . Now,we plan to record the video files with hbase. There are two problems with the hbase:
> 1 the video file is too large (100M avg.)
Why do you say that? The client side limit by default is -1
(disabled), and if there's any other limit they are just set
arbitrarily.
> 2 we require the transcoded video must can be read while it was been writing
That sounds like an application-level problem, not HBase.
>
> So,can anybody give some suggestions, Thanks.
>
> 2011-10-31
>
>
>
> regards
> xtliwen
>