You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Aditya Kumar <ad...@yahoo.com> on 2011/11/25 02:10:49 UTC
Amazon s3 and Hadoop/Hbase
Hi,
I am looking at Amazon S3.
is it like HDFS/Hadoop? .
If it is like Hadoop can we add only data to the tables.. than can we use Hbase?
I tried to read online,
all I see that S3 will not delete the data and will do versioing (something similar to Hadoop)
TIA,
Adi
Re: Amazon s3 and Hadoop/Hbase
Posted by Aditya Kumar <ad...@yahoo.com>.
Thanks Ram.
More details:
I want to keep the files for now.
when you say Analyses what type of analysis need HBase?
Can you please mention them..
As I am ina very early state I might not need any other thing than Keeping, BUT if it is needed for Analysies can you please let me know what it is >
________________________________
From: Ramkrishna S Vasudevan <ra...@huawei.com>
To: dev@hbase.apache.org; 'Aditya Kumar' <ad...@yahoo.com>
Sent: Thursday, November 24, 2011 9:07 PM
Subject: RE: Amazon s3 and Hadoop/Hbase
Hi
If your use case is like just store some files particularly if you need only
a big storage medium then S3 should be fine.
HBase is a database where you want your data to be added from some specific
source like a social site and you want to do some analysis.
Regards
Ram
-----Original Message-----
From: Aditya Kumar [mailto:adityakumar345@yahoo.com]
Sent: Friday, November 25, 2011 9:26 AM
To: dev@hbase.apache.org
Subject: Re: Amazon s3 and Hadoop/Hbase
Thanks Sonal for the excellent reply.
I am looking for a use case:
where in I want to store 10000 files and they will be accessed and modified
and kept back in the S3.
so there will be modifications of Files on S3 and trying to access them.
For a use case like this do I need Hbase on S3? (or) if I just use S3,
create bucket and keep on adding files and modify them on the need bases
________________________________
From: Sonal Goyal <so...@gmail.com>
To: dev@hbase.apache.org; Aditya Kumar <ad...@yahoo.com>
Sent: Thursday, November 24, 2011 7:11 PM
Subject: Re: Amazon s3 and Hadoop/Hbase
Aditya,
S3 is a file storage service. It does provide versioning and key based file
access, but it does not provide any processing paradigm, like Map Reduce
for Hadoop. HBase provides random access to data - its more like a database
in that you can have a key based access for records, and it is a columnar
store. S3 can be compared with HDFS, the hadoop distributed file system,
and in fact one can spin Hadoop clusters off S3 file system.
Maybe if you explain your problem in detail, we can help you further on the
mailing list. You can also read up the fundamentals on the Hadoop and HBase
wiki, there is a high level introduction which can clear the basic doubts.
Hope that helps.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar
<ad...@yahoo.com>wrote:
> Thanks Li Pi.
>
> so,
> does it mean that s3 is hadoop?
> can you please let me know what I will be missing if I use s3 only with
> out Hbase?
>
> 2) In s3 there is delete operation, does it really delete the key and
> there by value?
>
>
>
> ________________________________
> From: Li Pi <li...@idle.li>
> To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
> Sent: Thursday, November 24, 2011 5:29 PM
> Subject: Re: Amazon s3 and Hadoop/Hbase
>
> You can run hbase against s3.
> On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
>
> > Hi,
> > I am looking at Amazon S3.
> > is it like HDFS/Hadoop? .
> > If it is like Hadoop can we add only data to the tables.. than can we
use
> > Hbase?
> > I tried to read online,
> > all I see that S3 will not delete the data and will do versioing
> > (something similar to Hadoop)
> >
> > TIA,
> > Adi
>
Re: Amazon s3 and Hadoop/Hbase
Posted by Sonal Goyal <so...@gmail.com>.
If your use case is pure file modification, I am not sure how HBase is
helping you. Do you want to save some metadata as well? Are you doing some
file analysis and want to access the results? If not, you are probably
better off not adding another layer.
This is not to say HBase can not save file blobs. Just that in your
scenario, you dont seem to need the value HBase adds.
Thats my point of view! Lets see if someone on the list has some other
thoughts.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Fri, Nov 25, 2011 at 9:25 AM, Aditya Kumar <ad...@yahoo.com>wrote:
> Thanks Sonal for the excellent reply.
>
> I am looking for a use case:
> where in I want to store 10000 files and they will be accessed and
> modified and kept back in the S3.
>
> so there will be modifications of Files on S3 and trying to access them.
>
> For a use case like this do I need Hbase on S3? (or) if I just use S3,
> create bucket and keep on adding files and modify them on the need bases
>
>
> ________________________________
> From: Sonal Goyal <so...@gmail.com>
> To: dev@hbase.apache.org; Aditya Kumar <ad...@yahoo.com>
> Sent: Thursday, November 24, 2011 7:11 PM
> Subject: Re: Amazon s3 and Hadoop/Hbase
>
> Aditya,
>
> S3 is a file storage service. It does provide versioning and key based file
> access, but it does not provide any processing paradigm, like Map Reduce
> for Hadoop. HBase provides random access to data - its more like a database
> in that you can have a key based access for records, and it is a columnar
> store. S3 can be compared with HDFS, the hadoop distributed file system,
> and in fact one can spin Hadoop clusters off S3 file system.
>
> Maybe if you explain your problem in detail, we can help you further on the
> mailing list. You can also read up the fundamentals on the Hadoop and HBase
> wiki, there is a high level introduction which can clear the basic doubts.
>
> Hope that helps.
>
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar <adityakumar345@yahoo.com
> >wrote:
>
> > Thanks Li Pi.
> >
> > so,
> > does it mean that s3 is hadoop?
> > can you please let me know what I will be missing if I use s3 only with
> > out Hbase?
> >
> > 2) In s3 there is delete operation, does it really delete the key and
> > there by value?
> >
> >
> >
> > ________________________________
> > From: Li Pi <li...@idle.li>
> > To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
> > Sent: Thursday, November 24, 2011 5:29 PM
> > Subject: Re: Amazon s3 and Hadoop/Hbase
> >
> > You can run hbase against s3.
> > On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com>
> wrote:
> >
> > > Hi,
> > > I am looking at Amazon S3.
> > > is it like HDFS/Hadoop? .
> > > If it is like Hadoop can we add only data to the tables.. than can we
> use
> > > Hbase?
> > > I tried to read online,
> > > all I see that S3 will not delete the data and will do versioing
> > > (something similar to Hadoop)
> > >
> > > TIA,
> > > Adi
> >
>
RE: Amazon s3 and Hadoop/Hbase
Posted by Ramkrishna S Vasudevan <ra...@huawei.com>.
Hi
If your use case is like just store some files particularly if you need only
a big storage medium then S3 should be fine.
HBase is a database where you want your data to be added from some specific
source like a social site and you want to do some analysis.
Regards
Ram
-----Original Message-----
From: Aditya Kumar [mailto:adityakumar345@yahoo.com]
Sent: Friday, November 25, 2011 9:26 AM
To: dev@hbase.apache.org
Subject: Re: Amazon s3 and Hadoop/Hbase
Thanks Sonal for the excellent reply.
I am looking for a use case:
where in I want to store 10000 files and they will be accessed and modified
and kept back in the S3.
so there will be modifications of Files on S3 and trying to access them.
For a use case like this do I need Hbase on S3? (or) if I just use S3,
create bucket and keep on adding files and modify them on the need bases
________________________________
From: Sonal Goyal <so...@gmail.com>
To: dev@hbase.apache.org; Aditya Kumar <ad...@yahoo.com>
Sent: Thursday, November 24, 2011 7:11 PM
Subject: Re: Amazon s3 and Hadoop/Hbase
Aditya,
S3 is a file storage service. It does provide versioning and key based file
access, but it does not provide any processing paradigm, like Map Reduce
for Hadoop. HBase provides random access to data - its more like a database
in that you can have a key based access for records, and it is a columnar
store. S3 can be compared with HDFS, the hadoop distributed file system,
and in fact one can spin Hadoop clusters off S3 file system.
Maybe if you explain your problem in detail, we can help you further on the
mailing list. You can also read up the fundamentals on the Hadoop and HBase
wiki, there is a high level introduction which can clear the basic doubts.
Hope that helps.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar
<ad...@yahoo.com>wrote:
> Thanks Li Pi.
>
> so,
> does it mean that s3 is hadoop?
> can you please let me know what I will be missing if I use s3 only with
> out Hbase?
>
> 2) In s3 there is delete operation, does it really delete the key and
> there by value?
>
>
>
> ________________________________
> From: Li Pi <li...@idle.li>
> To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
> Sent: Thursday, November 24, 2011 5:29 PM
> Subject: Re: Amazon s3 and Hadoop/Hbase
>
> You can run hbase against s3.
> On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
>
> > Hi,
> > I am looking at Amazon S3.
> > is it like HDFS/Hadoop? .
> > If it is like Hadoop can we add only data to the tables.. than can we
use
> > Hbase?
> > I tried to read online,
> > all I see that S3 will not delete the data and will do versioing
> > (something similar to Hadoop)
> >
> > TIA,
> > Adi
>
Re: Amazon s3 and Hadoop/Hbase
Posted by Aditya Kumar <ad...@yahoo.com>.
Thanks Sonal for the excellent reply.
I am looking for a use case:
where in I want to store 10000 files and they will be accessed and modified and kept back in the S3.
so there will be modifications of Files on S3 and trying to access them.
For a use case like this do I need Hbase on S3? (or) if I just use S3, create bucket and keep on adding files and modify them on the need bases
________________________________
From: Sonal Goyal <so...@gmail.com>
To: dev@hbase.apache.org; Aditya Kumar <ad...@yahoo.com>
Sent: Thursday, November 24, 2011 7:11 PM
Subject: Re: Amazon s3 and Hadoop/Hbase
Aditya,
S3 is a file storage service. It does provide versioning and key based file
access, but it does not provide any processing paradigm, like Map Reduce
for Hadoop. HBase provides random access to data - its more like a database
in that you can have a key based access for records, and it is a columnar
store. S3 can be compared with HDFS, the hadoop distributed file system,
and in fact one can spin Hadoop clusters off S3 file system.
Maybe if you explain your problem in detail, we can help you further on the
mailing list. You can also read up the fundamentals on the Hadoop and HBase
wiki, there is a high level introduction which can clear the basic doubts.
Hope that helps.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar <ad...@yahoo.com>wrote:
> Thanks Li Pi.
>
> so,
> does it mean that s3 is hadoop?
> can you please let me know what I will be missing if I use s3 only with
> out Hbase?
>
> 2) In s3 there is delete operation, does it really delete the key and
> there by value?
>
>
>
> ________________________________
> From: Li Pi <li...@idle.li>
> To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
> Sent: Thursday, November 24, 2011 5:29 PM
> Subject: Re: Amazon s3 and Hadoop/Hbase
>
> You can run hbase against s3.
> On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
>
> > Hi,
> > I am looking at Amazon S3.
> > is it like HDFS/Hadoop? .
> > If it is like Hadoop can we add only data to the tables.. than can we use
> > Hbase?
> > I tried to read online,
> > all I see that S3 will not delete the data and will do versioing
> > (something similar to Hadoop)
> >
> > TIA,
> > Adi
>
Re: Amazon s3 and Hadoop/Hbase
Posted by Sonal Goyal <so...@gmail.com>.
Aditya,
S3 is a file storage service. It does provide versioning and key based file
access, but it does not provide any processing paradigm, like Map Reduce
for Hadoop. HBase provides random access to data - its more like a database
in that you can have a key based access for records, and it is a columnar
store. S3 can be compared with HDFS, the hadoop distributed file system,
and in fact one can spin Hadoop clusters off S3 file system.
Maybe if you explain your problem in detail, we can help you further on the
mailing list. You can also read up the fundamentals on the Hadoop and HBase
wiki, there is a high level introduction which can clear the basic doubts.
Hope that helps.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar <ad...@yahoo.com>wrote:
> Thanks Li Pi.
>
> so,
> does it mean that s3 is hadoop?
> can you please let me know what I will be missing if I use s3 only with
> out Hbase?
>
> 2) In s3 there is delete operation, does it really delete the key and
> there by value?
>
>
>
> ________________________________
> From: Li Pi <li...@idle.li>
> To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
> Sent: Thursday, November 24, 2011 5:29 PM
> Subject: Re: Amazon s3 and Hadoop/Hbase
>
> You can run hbase against s3.
> On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
>
> > Hi,
> > I am looking at Amazon S3.
> > is it like HDFS/Hadoop? .
> > If it is like Hadoop can we add only data to the tables.. than can we use
> > Hbase?
> > I tried to read online,
> > all I see that S3 will not delete the data and will do versioing
> > (something similar to Hadoop)
> >
> > TIA,
> > Adi
>
Re: Amazon s3 and Hadoop/Hbase
Posted by Li Pi <li...@idle.li>.
Hbase can run against any filesystem, not just hdfs. This includes local
filesystems like ext3. S3 is merely another distributed filesystem to hbase.
On Nov 24, 2011 5:40 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
> Thanks Li Pi.
>
> so,
> does it mean that s3 is hadoop?
> can you please let me know what I will be missing if I use s3 only with
> out Hbase?
>
> 2) In s3 there is delete operation, does it really delete the key and
> there by value?
>
>
>
> ________________________________
> From: Li Pi <li...@idle.li>
> To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
> Sent: Thursday, November 24, 2011 5:29 PM
> Subject: Re: Amazon s3 and Hadoop/Hbase
>
> You can run hbase against s3.
> On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
>
> > Hi,
> > I am looking at Amazon S3.
> > is it like HDFS/Hadoop? .
> > If it is like Hadoop can we add only data to the tables.. than can we use
> > Hbase?
> > I tried to read online,
> > all I see that S3 will not delete the data and will do versioing
> > (something similar to Hadoop)
> >
> > TIA,
> > Adi
Re: Amazon s3 and Hadoop/Hbase
Posted by Aditya Kumar <ad...@yahoo.com>.
Thanks Li Pi.
so,
does it mean that s3 is hadoop?
can you please let me know what I will be missing if I use s3 only with out Hbase?
2) In s3 there is delete operation, does it really delete the key and there by value?
________________________________
From: Li Pi <li...@idle.li>
To: Aditya Kumar <ad...@yahoo.com>; dev@hbase.apache.org
Sent: Thursday, November 24, 2011 5:29 PM
Subject: Re: Amazon s3 and Hadoop/Hbase
You can run hbase against s3.
On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
> Hi,
> I am looking at Amazon S3.
> is it like HDFS/Hadoop? .
> If it is like Hadoop can we add only data to the tables.. than can we use
> Hbase?
> I tried to read online,
> all I see that S3 will not delete the data and will do versioing
> (something similar to Hadoop)
>
> TIA,
> Adi
Re: Amazon s3 and Hadoop/Hbase
Posted by Li Pi <li...@idle.li>.
You can run hbase against s3.
On Nov 24, 2011 5:11 PM, "Aditya Kumar" <ad...@yahoo.com> wrote:
> Hi,
> I am looking at Amazon S3.
> is it like HDFS/Hadoop? .
> If it is like Hadoop can we add only data to the tables.. than can we use
> Hbase?
> I tried to read online,
> all I see that S3 will not delete the data and will do versioing
> (something similar to Hadoop)
>
> TIA,
> Adi