You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Buğra Çakır <bu...@oranteknoloji.com> on 2015/04/30 15:35:27 UTC
HBase Filesystem Adapter
Hi,
I would like to use HBase with distributed filesystems other
than HDFS. Are there any plans for developing filesystem
adapters for these distributed filesystems ? (ceph, glusterfs, ...)
Best,
Bugra
Re: HBase Filesystem Adapter
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at HBASE-13485
Cheers
> On Apr 30, 2015, at 6:35 AM, Buğra Çakır <bu...@oranteknoloji.com> wrote:
>
> Hi,
>
>
> I would like to use HBase with distributed filesystems other
>
> than HDFS. Are there any plans for developing filesystem
>
> adapters for these distributed filesystems ? (ceph, glusterfs, ...)
>
>
> Best,
>
> Bugra
>
>
Re: HBase Filesystem Adapter
Posted by Enis Söztutar <en...@apache.org>.
Here are my notes in raw form:
https://docs.google.com/document/d/14rpa1efWDMyMPZNqsDUD_skfVGtmUiOq7UkO9mEHHkw/edit?usp=sharing
Enis
On Thu, Apr 30, 2015 at 2:27 PM, Sean Busbey <bu...@cloudera.com> wrote:
> On Apr 30, 2015 4:11 PM, "Enis Söztutar" <en...@apache.org> wrote:
> >
> > This is a nice topic. Let's put it on the ref guide.
> >
> > Hbase on Azure FS is GA, and there has already been some work for
> > supporting HBase on the Hadoop native driver.
> > From this thread, my gathering is that, HBase should run on HDFS, MaprFS,
> > IBM GPFS, Azure WASB (and maybe Isilon, etc).
> >
> > I had a old write up of all the interaction points between HBase and the
> > underlying FS for semantic guarantees (atomic namespace rename, recover
> > lease, sync / flush ) etc. If anyone interested, I can try to dig it up.
> >
>
> Yes, please do! We can see about getting some of it added to the FileSystem
> spec Hadoop has been trying to get off the ground.
>
> -Sean
>
Re: HBase Filesystem Adapter
Posted by Sean Busbey <bu...@cloudera.com>.
On Apr 30, 2015 4:11 PM, "Enis Söztutar" <en...@apache.org> wrote:
>
> This is a nice topic. Let's put it on the ref guide.
>
> Hbase on Azure FS is GA, and there has already been some work for
> supporting HBase on the Hadoop native driver.
> From this thread, my gathering is that, HBase should run on HDFS, MaprFS,
> IBM GPFS, Azure WASB (and maybe Isilon, etc).
>
> I had a old write up of all the interaction points between HBase and the
> underlying FS for semantic guarantees (atomic namespace rename, recover
> lease, sync / flush ) etc. If anyone interested, I can try to dig it up.
>
Yes, please do! We can see about getting some of it added to the FileSystem
spec Hadoop has been trying to get off the ground.
-Sean
Re: HBase Filesystem Adapter
Posted by Enis Söztutar <en...@apache.org>.
This is a nice topic. Let's put it on the ref guide.
Hbase on Azure FS is GA, and there has already been some work for
supporting HBase on the Hadoop native driver.
>From this thread, my gathering is that, HBase should run on HDFS, MaprFS,
IBM GPFS, Azure WASB (and maybe Isilon, etc).
I had a old write up of all the interaction points between HBase and the
underlying FS for semantic guarantees (atomic namespace rename, recover
lease, sync / flush ) etc. If anyone interested, I can try to dig it up.
Enis
On Thu, Apr 30, 2015 at 1:46 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> I believe HBase also runs directly against Azure Blob Storage. This article
> [0] gives some details; I'm not sure if it's hit GA yet.
>
> -n
>
> [0]:
>
> http://azure.microsoft.com/blog/2014/06/06/azure-hdinsight-previewing-hbase-clusters-as-a-nosql-database-on-azure-blobs/
>
> On Thu, Apr 30, 2015 at 11:46 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > This thread is starting to sound like a new section for the ref guide. :)
> >
> > --
> > Sean
> > On Apr 30, 2015 1:07 PM, "Jerry He" <je...@gmail.com> wrote:
> >
> > > We've also made HBase running on IBM GPFS.
> > > http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System
> > >
> > > We have a Hadoop FileSystem implementation that translates hadoop calls
> > > into GPFS native calls.
> > > Overall it has been running well on live clusters.
> > >
> > > Jerry
> > >
> >
>
Re: HBase Filesystem Adapter
Posted by Nick Dimiduk <nd...@gmail.com>.
I believe HBase also runs directly against Azure Blob Storage. This article
[0] gives some details; I'm not sure if it's hit GA yet.
-n
[0]:
http://azure.microsoft.com/blog/2014/06/06/azure-hdinsight-previewing-hbase-clusters-as-a-nosql-database-on-azure-blobs/
On Thu, Apr 30, 2015 at 11:46 AM, Sean Busbey <bu...@cloudera.com> wrote:
> This thread is starting to sound like a new section for the ref guide. :)
>
> --
> Sean
> On Apr 30, 2015 1:07 PM, "Jerry He" <je...@gmail.com> wrote:
>
> > We've also made HBase running on IBM GPFS.
> > http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System
> >
> > We have a Hadoop FileSystem implementation that translates hadoop calls
> > into GPFS native calls.
> > Overall it has been running well on live clusters.
> >
> > Jerry
> >
>
Re: HBase Filesystem Adapter
Posted by Sean Busbey <bu...@cloudera.com>.
This thread is starting to sound like a new section for the ref guide. :)
--
Sean
On Apr 30, 2015 1:07 PM, "Jerry He" <je...@gmail.com> wrote:
> We've also made HBase running on IBM GPFS.
> http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System
>
> We have a Hadoop FileSystem implementation that translates hadoop calls
> into GPFS native calls.
> Overall it has been running well on live clusters.
>
> Jerry
>
Re: HBase Filesystem Adapter
Posted by Jerry He <je...@gmail.com>.
We've also made HBase running on IBM GPFS.
http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System
We have a Hadoop FileSystem implementation that translates hadoop calls
into GPFS native calls.
Overall it has been running well on live clusters.
Jerry
Re: HBase Filesystem Adapter
Posted by Sean Busbey <bu...@cloudera.com>.
On Thu, Apr 30, 2015 at 10:19 AM, Stack <st...@duboce.net> wrote:
> On Thu, Apr 30, 2015 at 6:35 AM, Buğra Çakır <
> bugra.cakir@oranteknoloji.com>
> wrote:
>
> > Hi,
> >
> >
> > I would like to use HBase with distributed filesystems other
> >
> > than HDFS. Are there any plans for developing filesystem
> >
> > adapters for these distributed filesystems ? (ceph, glusterfs, ...)
> >
> >
> What are you looking for in particular Bugra?
>
> There have been various attempts at running hbase over filesystems other
> than HDFS. HBase for the most part makes use of the Hadoop Filesystem
> Interface and has been reported out in the wild as running on other
> filesystems (S3?, MapR, and so on) with attendant compromises and benefit.
>
>
Gluster already has a Hadoop FileSystem implementation[1], but I haven't
seen it in practice.
EMC Isilon has a Hadoop FileSystem implementation as well that I have seen
HBase running on top of.
Essentially, all of these integrations happen outside of HBase. The
drawback is that there are places where we do checks for
DistributedFileSystem to make optimizations. However, those should all be
done in a way that falls back gracefully (not least because we need that in
order to support multiple HDFS versions). I'm not aware of us making
similar optimizations for other implementations, but I think we'd be fine
helping things along if a contributor showed up.
[1]: https://forge.gluster.org/hadoop/pages/Home
--
Sean
Re: HBase Filesystem Adapter
Posted by Michael Segel <mi...@hotmail.com>.
If you’re not going to be using HDFS and Map/Reduce, I would suggest you choose a different noSQL persistent data store.
> On May 1, 2015, at 6:49 AM, Buğra Çakır <bu...@oranteknoloji.com> wrote:
>
> Hi,
>
> I would like to use HBase in areas when we don't need functionality given within full stack Hadoop ecosystem So that's why I like to integrate HBase with the other dfs and I'm planning to dig into this :)
>
> Bugra
>
>
> ________________________________________
> Gönderen: saint.ack@gmail.com <sa...@gmail.com> adına Stack <st...@duboce.net>
> Gönderildi: 30 Nisan 2015 Perşembe 18:19
> Kime: Hbase-User
> Konu: Re: HBase Filesystem Adapter
>
> On Thu, Apr 30, 2015 at 6:35 AM, Buğra Çakır <bu...@oranteknoloji.com>
> wrote:
>
>> Hi,
>>
>>
>> I would like to use HBase with distributed filesystems other
>>
>> than HDFS. Are there any plans for developing filesystem
>>
>> adapters for these distributed filesystems ? (ceph, glusterfs, ...)
>>
>>
> What are you looking for in particular Bugra?
>
> There have been various attempts at running hbase over filesystems other
> than HDFS. HBase for the most part makes use of the Hadoop Filesystem
> Interface and has been reported out in the wild as running on other
> filesystems (S3?, MapR, and so on) with attendant compromises and benefit.
>
> I know of no current efforts at making hbase run on ceph, for instance
> (Would be very interested if such an effort were afoot).
>
> Thanks,
> St.Ack
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Ynt: HBase Filesystem Adapter
Posted by Buğra Çakır <bu...@oranteknoloji.com>.
Hi,
I would like to use HBase in areas when we don't need functionality given within full stack Hadoop ecosystem So that's why I like to integrate HBase with the other dfs and I'm planning to dig into this :)
Bugra
________________________________________
Gönderen: saint.ack@gmail.com <sa...@gmail.com> adına Stack <st...@duboce.net>
Gönderildi: 30 Nisan 2015 Perşembe 18:19
Kime: Hbase-User
Konu: Re: HBase Filesystem Adapter
On Thu, Apr 30, 2015 at 6:35 AM, Buğra Çakır <bu...@oranteknoloji.com>
wrote:
> Hi,
>
>
> I would like to use HBase with distributed filesystems other
>
> than HDFS. Are there any plans for developing filesystem
>
> adapters for these distributed filesystems ? (ceph, glusterfs, ...)
>
>
What are you looking for in particular Bugra?
There have been various attempts at running hbase over filesystems other
than HDFS. HBase for the most part makes use of the Hadoop Filesystem
Interface and has been reported out in the wild as running on other
filesystems (S3?, MapR, and so on) with attendant compromises and benefit.
I know of no current efforts at making hbase run on ceph, for instance
(Would be very interested if such an effort were afoot).
Thanks,
St.Ack
Re: HBase Filesystem Adapter
Posted by Stack <st...@duboce.net>.
On Thu, Apr 30, 2015 at 6:35 AM, Buğra Çakır <bu...@oranteknoloji.com>
wrote:
> Hi,
>
>
> I would like to use HBase with distributed filesystems other
>
> than HDFS. Are there any plans for developing filesystem
>
> adapters for these distributed filesystems ? (ceph, glusterfs, ...)
>
>
What are you looking for in particular Bugra?
There have been various attempts at running hbase over filesystems other
than HDFS. HBase for the most part makes use of the Hadoop Filesystem
Interface and has been reported out in the wild as running on other
filesystems (S3?, MapR, and so on) with attendant compromises and benefit.
I know of no current efforts at making hbase run on ceph, for instance
(Would be very interested if such an effort were afoot).
Thanks,
St.Ack
Re: HBase Filesystem Adapter
Posted by Michael Segel <mi...@hotmail.com>.
I would look at a different solution than HBase.
HBase works well because its tied closely to the HDFS and Hadoop ecosystem. Going outside of this… too many headaches and you’d be better off with a NoSQL engine like Cassandra or Riak, or something else.
> On Apr 30, 2015, at 8:35 AM, Buğra Çakır <bu...@oranteknoloji.com> wrote:
>
> Hi,
>
>
> I would like to use HBase with distributed filesystems other
>
> than HDFS. Are there any plans for developing filesystem
>
> adapters for these distributed filesystems ? (ceph, glusterfs, ...)
>
>
> Best,
>
> Bugra
>
>