You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Lars George <la...@gmail.com> on 2012/07/02 12:11:18 UTC
Powered By Page
Hi,
Please see http://wiki.apache.org/hadoop/Hbase/PoweredBy
Everyone on this list, kindly consider verifying that your entry on the Powered By page is current.
For those who are users of HBase but have not added yourself to the above page: if you are happy to share this with us and the rest of the world, please add your project.
Thank you all for helping out.
Regards,
Lars
Re: Powered By Page
Posted by Stack <st...@duboce.net>.
On Tue, Jul 3, 2012 at 1:29 PM, Buckley,Ron <bu...@oclc.org> wrote:
> Stack/Lars,
>
> Here's an entry for OCLC:
>
> OCLC (www.worldcat.org) uses HBase as the main data store for WorldCat,
> a union catalog which aggregates the collections of 72,000 libraries in
> 112 countries and territories. WorldCat is currently comprised of
> nearly 1 billion records with nearly 2 billion library ownership
> indications. We're running a 50 Node HBase cluster and a separate
> offline map-reduce cluster.
>
Done
(Are we allowed favorites? If so, this is one of mine...)
St.Ack
RE: Powered By Page
Posted by "Buckley,Ron" <bu...@oclc.org>.
Stack/Lars,
Here's an entry for OCLC:
OCLC (www.worldcat.org) uses HBase as the main data store for WorldCat,
a union catalog which aggregates the collections of 72,000 libraries in
112 countries and territories. WorldCat is currently comprised of
nearly 1 billion records with nearly 2 billion library ownership
indications. We're running a 50 Node HBase cluster and a separate
offline map-reduce cluster.
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Monday, July 02, 2012 8:13 AM
To: user@hbase.apache.org
Subject: Re: Powered By Page
On Mon, Jul 2, 2012 at 1:06 PM, Ben Cuthbert <be...@ymail.com>
wrote:
> Thanks Lars
>
> Just a note how do I edit and add? I have just registered.
>
>
Oh yeah... you have to be granted perms to edit wiki because it was
being spammed at a fierce rate... I should have remembered (might
explain why our powered-by page has gone stale).
Add text to this mail thread and I or Lars will hoist it up on to the
wiki? Unless ye have a better idea? (Check in the powered-by page
and let people make JIRA with patch? That seems to painful?)
St.Ack
Re: Powered By Page
Posted by Asaf Mesika <as...@gmail.com>.
Adding capcha didn't help?
Sent from my iPad
On 2 ביול 2012, at 15:13, Stack <st...@duboce.net> wrote:
> On Mon, Jul 2, 2012 at 1:06 PM, Ben Cuthbert <be...@ymail.com> wrote:
>> Thanks Lars
>>
>> Just a note how do I edit and add? I have just registered.
>>
>>
>
> Oh yeah... you have to be granted perms to edit wiki because it was
> being spammed at a fierce rate... I should have remembered (might
> explain why our powered-by page has gone stale).
>
> Add text to this mail thread and I or Lars will hoist it up on to the
> wiki? Unless ye have a better idea? (Check in the powered-by page
> and let people make JIRA with patch? That seems to painful?)
>
> St.Ack
Re: Powered By Page
Posted by Stack <st...@duboce.net>.
On Mon, Jul 2, 2012 at 11:16 PM, Taylor, Ronald C
<ro...@pnnl.gov> wrote:
...
>
Thanks for the interesting setup description Ronald. Definitely
interested in how things progress.
Go easy,
St.Ack
RE: Powered By Page
Posted by "Taylor, Ronald C" <ro...@pnnl.gov>.
Hi Stack,
Re Lustre use: I'm not a hardware infrastructure type of guy, but I can tell you that we have a very fast interconnect for access into the global filesystem:
"The Olympus Infiniband topology is a combination of 2:1 oversubscribed 36 port leaf switches and direct links into a 648 port core Qlogic QDR Infiniband switch."
I am not really worried about loss of data locality and slower speed of access to the Hbase tables. That is, this is not (yet) a production environment for multiple users with real time access. Though I think it would work - it's been quite stable, for one thing, and I have not noticed any speed problem in retrieving records. But I have not done any serious timings, and currently we are not stressing Hbase, in that the warehouse is being used by a just a few bioinformaticians, not the general community, so to speak. I'm happy to simply have the data gathered in one place that provides scalability and for which I can easily write custom analytics programs that I can build upon and that won't have to be moved to another database framework down the line.
As the warehouse grows, I do plan on doing some testing, comparing HBase access using local disk storage vs Lustre. But that's when I have more time, and the warehouse is large enough for some real testing. We also have the option of putting *everything* into Lustre, both Hbase tables and all temp HDFS file storage used by our Map Reduce programs. So - no local disk use at all. I'm curious as to how well that would work. Possibly quite well, but no testing yet. Want to try that. It should be a pretty simple switch - our olympus support people have already constructed alternate starting points that load all the libs into Lustre instead of each local disk), but got other more immediate work to do first.
BTW - the Dept of Energy's new five-year systems biology knowledgebase project - the largest single bioinformatics project at DOE, I believe - is using Hadoop for several things in its multiple backends. See http://kbase.science.energy.gov/. I believe that Michael Schatz at Cold Spring Harbor Lab is heading up the Hadoop work, with clusters at Lawrence Berkeley, Argonne Nat Lab, and Oak Ridge. Not sure how HBase fits in -they are getting into some NoSQL work, but not sure what they'll be using. HBase, I hope, but don't know.
Ron
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
Richland, WA 99352
phone: (509) 372-6568
email: ronald.taylor@pnnl.gov
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Monday, July 02, 2012 1:37 PM
To: user@hbase.apache.org
Subject: Re: Powered By Page
On Mon, Jul 2, 2012 at 8:19 PM, Taylor, Ronald C <ro...@pnnl.gov> wrote:
> Pacific Northwest National Laboratory (www.pnl.gov) - Hadoop and HBase (Cloudera distribution) are being used within PNNL's Computational Biology & Bioinformatics Group for a systems biology data warehouse project that integrates high throughput proteomics and transcriptomics data sets coming from instruments in the Environmental Molecular Sciences Laboratory, a US Department of Energy national user facility located at PNNL. The data sets are being merged and annotated with other public genomics information in the data warehouse environment, with Hadoop analysis programs operating on the annotated data in the HBase tables. This work is hosted by olympus, a large PNNL institutional computing cluster (http://www.pnl.gov/news/release.aspx?id=908) , with the HBase tables being stored in olympus's Lustre file system.
>
Thats a cool one. I put it up (I put it in place of the powerset entry -- smile).
How's that Lustre hookup work Ronald? You did your own FS implementation for it?
Good stuff,
Thanks.
St.Ack
Re: Powered By Page
Posted by Stack <st...@duboce.net>.
On Mon, Jul 2, 2012 at 8:19 PM, Taylor, Ronald C <ro...@pnnl.gov> wrote:
> Pacific Northwest National Laboratory (www.pnl.gov) - Hadoop and HBase (Cloudera distribution) are being used within PNNL's Computational Biology & Bioinformatics Group for a systems biology data warehouse project that integrates high throughput proteomics and transcriptomics data sets coming from instruments in the Environmental Molecular Sciences Laboratory, a US Department of Energy national user facility located at PNNL. The data sets are being merged and annotated with other public genomics information in the data warehouse environment, with Hadoop analysis programs operating on the annotated data in the HBase tables. This work is hosted by olympus, a large PNNL institutional computing cluster (http://www.pnl.gov/news/release.aspx?id=908) , with the HBase tables being stored in olympus's Lustre file system.
>
Thats a cool one. I put it up (I put it in place of the powerset
entry -- smile).
How's that Lustre hookup work Ronald? You did your own FS
implementation for it?
Good stuff,
Thanks.
St.Ack
RE: Powered By Page
Posted by "Taylor, Ronald C" <ro...@pnnl.gov>.
Hello Stack, Lars,
Here is a scientific application, if you want to add it:
Pacific Northwest National Laboratory (www.pnl.gov) - Hadoop and HBase (Cloudera distribution) are being used within PNNL's Computational Biology & Bioinformatics Group for a systems biology data warehouse project that integrates high throughput proteomics and transcriptomics data sets coming from instruments in the Environmental Molecular Sciences Laboratory, a US Department of Energy national user facility located at PNNL. The data sets are being merged and annotated with other public genomics information in the data warehouse environment, with Hadoop analysis programs operating on the annotated data in the HBase tables. This work is hosted by olympus, a large PNNL institutional computing cluster (http://www.pnl.gov/news/release.aspx?id=908) , with the HBase tables being stored in olympus's Lustre file system.
Cheers,
Ron
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
Richland, WA 99352
phone: (509) 372-6568
email: ronald.taylor@pnnl.gov
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Monday, July 02, 2012 5:13 AM
To: user@hbase.apache.org
Subject: Re: Powered By Page
On Mon, Jul 2, 2012 at 1:06 PM, Ben Cuthbert <be...@ymail.com> wrote:
> Thanks Lars
>
> Just a note how do I edit and add? I have just registered.
>
>
Oh yeah... you have to be granted perms to edit wiki because it was being spammed at a fierce rate... I should have remembered (might explain why our powered-by page has gone stale).
Add text to this mail thread and I or Lars will hoist it up on to the wiki? Unless ye have a better idea? (Check in the powered-by page and let people make JIRA with patch? That seems to painful?)
St.Ack
Re: Powered By Page
Posted by Stack <st...@duboce.net>.
On Mon, Jul 2, 2012 at 1:06 PM, Ben Cuthbert <be...@ymail.com> wrote:
> Thanks Lars
>
> Just a note how do I edit and add? I have just registered.
>
>
Oh yeah... you have to be granted perms to edit wiki because it was
being spammed at a fierce rate... I should have remembered (might
explain why our powered-by page has gone stale).
Add text to this mail thread and I or Lars will hoist it up on to the
wiki? Unless ye have a better idea? (Check in the powered-by page
and let people make JIRA with patch? That seems to painful?)
St.Ack
Re: Powered By Page
Posted by Ben Cuthbert <be...@ymail.com>.
Thanks Lars
Just a note how do I edit and add? I have just registered.
On 2 Jul 2012, at 11:50, Ulrich Staudinger wrote:
> Cheers everyone,
> I tried already adding mine, but the page says "Immutable Page"
>
> Why is it immutable?
>
> Thanks
>
>
> On Mon, Jul 2, 2012 at 12:32 PM, Lars George <la...@gmail.com> wrote:
>
>> Hi Ben,
>>
>> Please do so, you can create yourself an account and edit the Wiki page.
>> Let us know if you get stuck.
>>
>> Thanks for sharing!
>>
>> Lars
>>
>> On Jul 2, 2012, at 12:20 PM, Ben Cuthbert wrote:
>>
>>> Hi Lars
>>>
>>> We would love to add our company
>>>
>>> http://www.celer-tech.com
>>>
>>> Regards
>>>
>>> Ben
>>> On 2 Jul 2012, at 11:11, Lars George wrote:
>>>
>>>> Hi,
>>>>
>>>> Please see http://wiki.apache.org/hadoop/Hbase/PoweredBy
>>>>
>>>> Everyone on this list, kindly consider verifying that your entry on the
>> Powered By page is current.
>>>>
>>>> For those who are users of HBase but have not added yourself to the
>> above page: if you are happy to share this with us and the rest of the
>> world, please add your project.
>>>>
>>>> Thank you all for helping out.
>>>>
>>>> Regards,
>>>> Lars
>>>
>>
>>
>
>
> --
> Ulrich Staudinger
>
> <http://goog_958005736>http://www.activequant.com
> Connect online: https://www.xing.com/profile/Ulrich_Staudinger
Re: Powered By Page
Posted by Ulrich Staudinger <us...@activequant.com>.
Cheers everyone,
I tried already adding mine, but the page says "Immutable Page"
Why is it immutable?
Thanks
On Mon, Jul 2, 2012 at 12:32 PM, Lars George <la...@gmail.com> wrote:
> Hi Ben,
>
> Please do so, you can create yourself an account and edit the Wiki page.
> Let us know if you get stuck.
>
> Thanks for sharing!
>
> Lars
>
> On Jul 2, 2012, at 12:20 PM, Ben Cuthbert wrote:
>
> > Hi Lars
> >
> > We would love to add our company
> >
> > http://www.celer-tech.com
> >
> > Regards
> >
> > Ben
> > On 2 Jul 2012, at 11:11, Lars George wrote:
> >
> >> Hi,
> >>
> >> Please see http://wiki.apache.org/hadoop/Hbase/PoweredBy
> >>
> >> Everyone on this list, kindly consider verifying that your entry on the
> Powered By page is current.
> >>
> >> For those who are users of HBase but have not added yourself to the
> above page: if you are happy to share this with us and the rest of the
> world, please add your project.
> >>
> >> Thank you all for helping out.
> >>
> >> Regards,
> >> Lars
> >
>
>
--
Ulrich Staudinger
<http://goog_958005736>http://www.activequant.com
Connect online: https://www.xing.com/profile/Ulrich_Staudinger
Re: Powered By Page
Posted by Lars George <la...@gmail.com>.
Hi Ben,
Please do so, you can create yourself an account and edit the Wiki page. Let us know if you get stuck.
Thanks for sharing!
Lars
On Jul 2, 2012, at 12:20 PM, Ben Cuthbert wrote:
> Hi Lars
>
> We would love to add our company
>
> http://www.celer-tech.com
>
> Regards
>
> Ben
> On 2 Jul 2012, at 11:11, Lars George wrote:
>
>> Hi,
>>
>> Please see http://wiki.apache.org/hadoop/Hbase/PoweredBy
>>
>> Everyone on this list, kindly consider verifying that your entry on the Powered By page is current.
>>
>> For those who are users of HBase but have not added yourself to the above page: if you are happy to share this with us and the rest of the world, please add your project.
>>
>> Thank you all for helping out.
>>
>> Regards,
>> Lars
>
Re: Powered By Page
Posted by Ben Cuthbert <be...@ymail.com>.
Hi Lars
We would love to add our company
http://www.celer-tech.com
Regards
Ben
On 2 Jul 2012, at 11:11, Lars George wrote:
> Hi,
>
> Please see http://wiki.apache.org/hadoop/Hbase/PoweredBy
>
> Everyone on this list, kindly consider verifying that your entry on the Powered By page is current.
>
> For those who are users of HBase but have not added yourself to the above page: if you are happy to share this with us and the rest of the world, please add your project.
>
> Thank you all for helping out.
>
> Regards,
> Lars