You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Serega Sheypak <se...@gmail.com> on 2014/08/08 09:46:43 UTC
serving static images using HBase
Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
thumbs.
These images are used in many web-sites as static content.
Is it good Idea to place them to HBase and serve to users?
right now all images are on SSD discs with complicated partitioning and
backup procedures. It becomes hard to maintain them because qtty and total
size of images grow.
Re: serving static images using HBase
Posted by Serega Sheypak <se...@gmail.com>.
Thank you. I'm not "native" python writer, so I see that plugin reads and
writes from/to HBase through thrift-server, but I didn't explore
integration between thumbor.
Thanks again for the explanation.
2014-08-08 17:01 GMT+04:00 Damien Hardy <dh...@viadeoteam.com>:
> Yes it is.
> My plugin is just a storage/loader plugin for Thumbor using hbase (via
> thrift) default are filesystem, mongodb or redis.
>
> Thumbor can :
> * load image from multiple origins (filer, http, s3) or any loader
> plugin.
> * Store a copy in storage available (TTL can be implemented to force
> refresh)
> * Provide an HTTP API to transform images
> * optionally store results in storage
>
>
> Le 08/08/2014 14:47, Serega Sheypak a écrit :
> > Damien Hardy, is it true, that your plugin allows:
> > 1. store original images to hbase
> > 2. resize orignal images and persist result back to hbase?
> >
> >
> > 2014-08-08 12:00 GMT+04:00 Damien Hardy <dh...@viadeoteam.com>:
> >
> >> Hello Serega,
> >>
> >> We use it this way here via a python image manipulation service named
> >> Thumbor
> >> https://github.com/thumbor/thumbor/
> >> + a pluggin of my own :
> >>
> >>
> https://github.com/thumbor/thumbor/wiki/Plugins#thumbor_hbase-by-damien-hardy
> >>
> >> One big advantage is you can use it with lazy loading plugin HBase as
> >> storage and filer or http source as loader this way images will populate
> >> Hbase at the first call if missing in HBase.
> >>
> >> keys are spread on cluster with md5 prefix to avoid hotspot.
> >>
> >>
> >> Le 08/08/2014 09:46, Serega Sheypak a écrit :
> >>> Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
> >>> thumbs.
> >>> These images are used in many web-sites as static content.
> >>> Is it good Idea to place them to HBase and serve to users?
> >>>
> >>> right now all images are on SSD discs with complicated partitioning and
> >>> backup procedures. It becomes hard to maintain them because qtty and
> >> total
> >>> size of images grow.
> >>>
> >>
> >> --
> >> Damien HARDY
> >>
> >>
> >>
> >
>
> --
> Damien HARDY
> IT Infrastructure Architect
> Viadeo - 30 rue de la Victoire - 75009 Paris - France
> PGP : 45D7F89A
>
>
Re: serving static images using HBase
Posted by Damien Hardy <dh...@viadeoteam.com>.
Yes it is.
My plugin is just a storage/loader plugin for Thumbor using hbase (via
thrift) default are filesystem, mongodb or redis.
Thumbor can :
* load image from multiple origins (filer, http, s3) or any loader plugin.
* Store a copy in storage available (TTL can be implemented to force
refresh)
* Provide an HTTP API to transform images
* optionally store results in storage
Le 08/08/2014 14:47, Serega Sheypak a écrit :
> Damien Hardy, is it true, that your plugin allows:
> 1. store original images to hbase
> 2. resize orignal images and persist result back to hbase?
>
>
> 2014-08-08 12:00 GMT+04:00 Damien Hardy <dh...@viadeoteam.com>:
>
>> Hello Serega,
>>
>> We use it this way here via a python image manipulation service named
>> Thumbor
>> https://github.com/thumbor/thumbor/
>> + a pluggin of my own :
>>
>> https://github.com/thumbor/thumbor/wiki/Plugins#thumbor_hbase-by-damien-hardy
>>
>> One big advantage is you can use it with lazy loading plugin HBase as
>> storage and filer or http source as loader this way images will populate
>> Hbase at the first call if missing in HBase.
>>
>> keys are spread on cluster with md5 prefix to avoid hotspot.
>>
>>
>> Le 08/08/2014 09:46, Serega Sheypak a écrit :
>>> Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
>>> thumbs.
>>> These images are used in many web-sites as static content.
>>> Is it good Idea to place them to HBase and serve to users?
>>>
>>> right now all images are on SSD discs with complicated partitioning and
>>> backup procedures. It becomes hard to maintain them because qtty and
>> total
>>> size of images grow.
>>>
>>
>> --
>> Damien HARDY
>>
>>
>>
>
--
Damien HARDY
IT Infrastructure Architect
Viadeo - 30 rue de la Victoire - 75009 Paris - France
PGP : 45D7F89A
Re: serving static images using HBase
Posted by Serega Sheypak <se...@gmail.com>.
Damien Hardy, is it true, that your plugin allows:
1. store original images to hbase
2. resize orignal images and persist result back to hbase?
2014-08-08 12:00 GMT+04:00 Damien Hardy <dh...@viadeoteam.com>:
> Hello Serega,
>
> We use it this way here via a python image manipulation service named
> Thumbor
> https://github.com/thumbor/thumbor/
> + a pluggin of my own :
>
> https://github.com/thumbor/thumbor/wiki/Plugins#thumbor_hbase-by-damien-hardy
>
> One big advantage is you can use it with lazy loading plugin HBase as
> storage and filer or http source as loader this way images will populate
> Hbase at the first call if missing in HBase.
>
> keys are spread on cluster with md5 prefix to avoid hotspot.
>
>
> Le 08/08/2014 09:46, Serega Sheypak a écrit :
> > Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
> > thumbs.
> > These images are used in many web-sites as static content.
> > Is it good Idea to place them to HBase and serve to users?
> >
> > right now all images are on SSD discs with complicated partitioning and
> > backup procedures. It becomes hard to maintain them because qtty and
> total
> > size of images grow.
> >
>
> --
> Damien HARDY
>
>
>
Re: serving static images using HBase
Posted by Damien Hardy <dh...@viadeoteam.com>.
Hello Serega,
We use it this way here via a python image manipulation service named
Thumbor
https://github.com/thumbor/thumbor/
+ a pluggin of my own :
https://github.com/thumbor/thumbor/wiki/Plugins#thumbor_hbase-by-damien-hardy
One big advantage is you can use it with lazy loading plugin HBase as
storage and filer or http source as loader this way images will populate
Hbase at the first call if missing in HBase.
keys are spread on cluster with md5 prefix to avoid hotspot.
Le 08/08/2014 09:46, Serega Sheypak a écrit :
> Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
> thumbs.
> These images are used in many web-sites as static content.
> Is it good Idea to place them to HBase and serve to users?
>
> right now all images are on SSD discs with complicated partitioning and
> backup procedures. It becomes hard to maintain them because qtty and total
> size of images grow.
>
--
Damien HARDY
Re: serving static images using HBase
Posted by Serega Sheypak <se...@gmail.com>.
Thanks, look cool! We are going to test-drive it.
2014-08-08 12:09 GMT+04:00 Esteban Gutierrez <es...@cloudera.com>:
> Hi Serega,
>
> Yes, some have done it already:
> http://www.slideshare.net/jacque74/hug-hbase-presentation however there
> are
> caveats to consider: compactions, splits, sizes of the regions, etc.
>
> Currently there is work in progress in
> https://issues.apache.org/jira/browse/HBASE-11339 and that will help users
> to address this need to store medium size binary objects directly in HBase
> without the side effects of compactions and region splits.
>
> cheers,
> esteban.
>
>
>
>
> --
> Cloudera, Inc.
>
>
>
> On Fri, Aug 8, 2014 at 12:46 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
> > thumbs.
> > These images are used in many web-sites as static content.
> > Is it good Idea to place them to HBase and serve to users?
> >
> > right now all images are on SSD discs with complicated partitioning and
> > backup procedures. It becomes hard to maintain them because qtty and
> total
> > size of images grow.
> >
>
Re: serving static images using HBase
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi Serega,
Yes, some have done it already:
http://www.slideshare.net/jacque74/hug-hbase-presentation however there are
caveats to consider: compactions, splits, sizes of the regions, etc.
Currently there is work in progress in
https://issues.apache.org/jira/browse/HBASE-11339 and that will help users
to address this need to store medium size binary objects directly in HBase
without the side effects of compactions and region splits.
cheers,
esteban.
--
Cloudera, Inc.
On Fri, Aug 8, 2014 at 12:46 AM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, we have 10^8 small jpg/png images each size is 20-100 Kb. They are
> thumbs.
> These images are used in many web-sites as static content.
> Is it good Idea to place them to HBase and serve to users?
>
> right now all images are on SSD discs with complicated partitioning and
> backup procedures. It becomes hard to maintain them because qtty and total
> size of images grow.
>