You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Fuad Efendi <fu...@efendi.ca> on 2011/08/01 00:10:15 UTC
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
What is it all about? HBase sucks. Too many problems to newcomers,
few-weeks-warm-up to begin with!!!!!!!!!!!!!!!! Is it really-really
supported by Microsoft employees?!
And, SEO of course:
===================
--
Fuad Efendi
416-993-2060
Tokenizer Inc., Canada
Data Mining, Search Engines
http://www.tokenizer.ca
On 11-07-29 7:49 PM, "Otis Gospodnetic" <ot...@yahoo.com> wrote:
>Hi,
>
>I'm for publishing all performance metrics in JMX (in addition to
>exposing it wherever else you guys decide). That's because JMX is
>probably the easiest for our SPM for HBase [1] to get to HBase
>performance metrics and I suspect we are not alone.
>
>Otis
>[1] http://sematext.com/spm/hbase-performance-monitoring/index.html
>----
>Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
>Hadoop ecosystem search :: http://search-hadoop.com/
>
>
>
>>________________________________
>>From: Andrew Purtell <ap...@apache.org>
>>To: Doug Meil <do...@explorysmedical.com>; "dev@hbase.apache.org"
>><de...@hbase.apache.org>
>>Sent: Friday, July 29, 2011 4:34 PM
>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>
>>> I'd rather see this output being able to be captured by something the
>>>sink that Todd suggested, rather than focusing on shell access.
>>
>>
>>I don't agree.
>>
>>
>>Look at what we have existing and proposed:
>>
>> - Java API access to server and region load information, that the
>>shell uses
>>
>> - A proposal to dump some stats into log files, that then has to be
>>scraped
>>
>> - A proposal (by the FB guys) to export some JSON via a HTTP servlet
>>
>>This is not good design, this is a bunch of random shit stuck together.
>>
>>Note that what Todd proposed does not preclude adding Java client API
>>support for retrieving it.
>>
>>At a minimum all of this information must be accessible via the Java
>>client API, to enable programmatic monitoring and analysis use cases.
>>I'll add the shell support if nobody else cares about it, that is a
>>relatively small detail, but one I think is important.
>>
>>Best regards,
>>
>>
>> - Andy
>>
>>
>>Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>(via Tom White)
>>
>>
>>>________________________________
>>>From: Doug Meil <do...@explorysmedical.com>
>>>To: "dev@hbase.apache.org" <de...@hbase.apache.org>;
>>>"apurtell@apache.org" <ap...@apache.org>
>>>Sent: Friday, July 29, 2011 11:39 AM
>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>
>>>
>>>I'd rather see this output being able to be captured by something the
>>>sink
>>>that Todd suggested, rather than focusing on shell access. HServerLoad
>>>is
>>>super-summary at the RS level, and both the items in 4089 and 4147 are
>>>proposed to be "summarized" but still have reasonable detail (e.g., even
>>>table/CF summary there could be dozens of entries given a reasonably
>>>complex system).
>>>
>>>
>>>
>>>
>>>On 7/29/11 1:15 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>>>
>>>>There is also the matter of HServerLoad and how that is used by the
>>>>shell
>>>>and master UI to report on cluster status.
>>>>
>>>>I'd like the shell to be able to let the user explore all of these
>>>>different reports interactively.
>>>>
>>>>At the very least, they should all be handled the same way.
>>>>
>>>>And then there is Riley's work over at FB on a slow query log. How does
>>>>that fit in?
>>>>
>>>>Best regards,
>>>>
>>>>
>>>> - Andy
>>>>
>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>Hein
>>>>(via Tom White)
>>>>
>>>>
>>>>>________________________________
>>>>>From: Todd Lipcon <to...@cloudera.com>
>>>>>To: dev@hbase.apache.org
>>>>>Sent: Friday, July 29, 2011 9:58 AM
>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>
>>>>>What I'd prefer is something like:
>>>>>
>>>>>interface BlockCacheReportSink {
>>>>> public void reportStats(BlockCacheReport report);
>>>>>}
>>>>>
>>>>>class LoggingBlockCacheReportSink {
>>>>> ... {
>>>>> log it with whatever formatting you want
>>>>> }
>>>>>}
>>>>>
>>>>>then a configuration which could default to the logging
>>>>>implementation,
>>>>>but
>>>>>orgs could easily substitute their own implementation. For example, I
>>>>>could
>>>>>see wanting to do an implementation where it keeps local RRD graphs of
>>>>>some
>>>>>stats, or pushes them to a central management server.
>>>>>
>>>>>The assumption is that BlockCacheReport is a fairly straightforward
>>>>>"struct"
>>>>>with the non-formatted information available.
>>>>>
>>>>>-Todd
>>>>>
>>>>>On Fri, Jul 29, 2011 at 4:15 AM, Doug Meil
>>>>><do...@explorysmedical.com>wrote:
>>>>>
>>>>>>
>>>>>> Hi Folks-
>>>>>>
>>>>>> You probably already my email yesterday on this...
>>>>>> https://issues.apache.org/jira/browse/HBASE-4089 (block cache
>>>>>>report)
>>>>>>
>>>>>> ...and I just created this one...
>>>>>> https://issues.apache.org/jira/browse/HBASE-4147 (StoreFile query
>>>>>> report)
>>>>>>
>>>>>> What I'd like to run past the dev-list is this: if Hbase had
>>>>>>periodic
>>>>>> summary usage statistics, where should they go? What I'd like to
>>>>>>throw
>>>>>> out for discussion is that I'm suggesting that it should simply go
>>>>>>to
>>>>>>the
>>>>>> log files and users can slice and dice this on their own. No UI
>>>>>>(I.e.,
>>>>>> JSPs), no JMX, etc.
>>>>>>
>>>>>>
>>>>>> The summary out the output is this:
>>>>>> BlockCacheReport: on configured interval, print out summary of
>>>>>>blockcache
>>>>>> (at table/CF level) to log file. This one is point-in-time, not
>>>>>>delta.
>>>>>>
>>>>>> StoreFile read report: on configured interval, print out summary of
>>>>>> StoreFile accesses and how much time was spent reading each
>>>>>>StoreFile
>>>>>>to
>>>>>> log file.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Doug
>>>>>>
>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>Todd Lipcon
>>>>>Software Engineer, Cloudera
>>>>>
>>>>>
>>>
>>>
>>>
>>>
>>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Andrew Purtell <ap...@apache.org>.
This is the first time I've ever seen a thread here go off into what the fuckville. Bookmarked.
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Doug Meil <do...@explorysmedical.com>
>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>Sent: Sunday, July 31, 2011 4:17 PM
>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>
>
>re: "Whirr"
>
>I'm sure Cloudera would love to hear your feedback on Whirr, but please
>address it to them directly and not on the Hbase dist-list.
>
>re: "Is it really-really supported by Microsoft employees?!"
>
>It is really, really not.
>
>
>As you pointed out, and as is cited in the Apache Hbase book, Hbase was
>inspired from BigTable. And as for the refund that Ryan suggested I'm
>sure Google would be happy to make good on it. Watch out for the exchange
>rate at the border.
>
>
>
>
>On 7/31/11 6:38 PM, "Fuad Efendi" <fu...@efendi.ca> wrote:
>
>>Great,
>>
>>
>>
>>My VP is former MS-98 :)))
>>
>>
>>
>>(BTW, many thanks to http://www.cloudera.com/ employees; please make Whirr
>>really "rrrrrŠŠŠ" "open source"? I don't see any meaningŠ reinventing a
>>bike is cheaper nowadays!!!)
>>
>>
>>
>>
>>
>>
>>And, SEO of course: (remember: Google is SMARTER!!! You are just "clone".)
>>
>>==========================================================================
>>
>>
>>--
>>Fuad Efendi
>>416-993-2060
>>Tokenizer Inc., Canada
>>Data Mining, Search Engines
>>http://www.tokenizer.ca
>>
>>
>>
>>
>>
>>
>>
>>
>>On 11-07-31 6:21 PM, "Ryan Rawson" <ry...@gmail.com> wrote:
>>
>>>You should ask for your money back!!
>>>
>>>On Sun, Jul 31, 2011 at 3:10 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>>> What is it all about? HBase sucks. Too many problems to newcomers,
>>>> few-weeks-warm-up to begin with!!!!!!!!!!!!!!!! Is it really-really
>>>> supported by Microsoft employees?!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> And, SEO of course:
>>>> ===================
>>>>
>>>>
>>>> --
>>>> Fuad Efendi
>>>> 416-993-2060
>>>> Tokenizer Inc., Canada
>>>> Data Mining, Search Engines
>>>> http://www.tokenizer.ca
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 11-07-29 7:49 PM, "Otis Gospodnetic" <ot...@yahoo.com>
>>>>wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>I'm for publishing all performance metrics in JMX (in addition to
>>>>>exposing it wherever else you guys decide). That's because JMX is
>>>>>probably the easiest for our SPM for HBase [1] to get to HBase
>>>>>performance metrics and I suspect we are not alone.
>>>>>
>>>>>Otis
>>>>>[1] http://sematext.com/spm/hbase-performance-monitoring/index.html
>>>>>----
>>>>>Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
>>>>>Hadoop ecosystem search :: http://search-hadoop.com/
>>>>>
>>>>>
>>>>>
>>>>>>________________________________
>>>>>>From: Andrew Purtell <ap...@apache.org>
>>>>>>To: Doug Meil <do...@explorysmedical.com>; "dev@hbase.apache.org"
>>>>>><de...@hbase.apache.org>
>>>>>>Sent: Friday, July 29, 2011 4:34 PM
>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>
>>>>>>> I'd rather see this output being able to be captured by something
>>>>>>>the
>>>>>>>sink that Todd suggested, rather than focusing on shell access.
>>>>>>
>>>>>>
>>>>>>I don't agree.
>>>>>>
>>>>>>
>>>>>>Look at what we have existing and proposed:
>>>>>>
>>>>>> - Java API access to server and region load information, that the
>>>>>>shell uses
>>>>>>
>>>>>> - A proposal to dump some stats into log files, that then has to
>>>>>>be
>>>>>>scraped
>>>>>>
>>>>>> - A proposal (by the FB guys) to export some JSON via a HTTP
>>>>>>servlet
>>>>>>
>>>>>>This is not good design, this is a bunch of random shit stuck
>>>>>>together.
>>>>>>
>>>>>>Note that what Todd proposed does not preclude adding Java client API
>>>>>>support for retrieving it.
>>>>>>
>>>>>>At a minimum all of this information must be accessible via the Java
>>>>>>client API, to enable programmatic monitoring and analysis use cases.
>>>>>>I'll add the shell support if nobody else cares about it, that is a
>>>>>>relatively small detail, but one I think is important.
>>>>>>
>>>>>>Best regards,
>>>>>>
>>>>>>
>>>>>> - Andy
>>>>>>
>>>>>>
>>>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>Hein
>>>>>>(via Tom White)
>>>>>>
>>>>>>
>>>>>>>________________________________
>>>>>>>From: Doug Meil <do...@explorysmedical.com>
>>>>>>>To: "dev@hbase.apache.org" <de...@hbase.apache.org>;
>>>>>>>"apurtell@apache.org" <ap...@apache.org>
>>>>>>>Sent: Friday, July 29, 2011 11:39 AM
>>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>>
>>>>>>>
>>>>>>>I'd rather see this output being able to be captured by something the
>>>>>>>sink
>>>>>>>that Todd suggested, rather than focusing on shell access.
>>>>>>>HServerLoad
>>>>>>>is
>>>>>>>super-summary at the RS level, and both the items in 4089 and 4147
>>>>>>>are
>>>>>>>proposed to be "summarized" but still have reasonable detail (e.g.,
>>>>>>>even
>>>>>>>table/CF summary there could be dozens of entries given a reasonably
>>>>>>>complex system).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>On 7/29/11 1:15 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>>>>>>>
>>>>>>>>There is also the matter of HServerLoad and how that is used by the
>>>>>>>>shell
>>>>>>>>and master UI to report on cluster status.
>>>>>>>>
>>>>>>>>I'd like the shell to be able to let the user explore all of these
>>>>>>>>different reports interactively.
>>>>>>>>
>>>>>>>>At the very least, they should all be handled the same way.
>>>>>>>>
>>>>>>>>And then there is Riley's work over at FB on a slow query log. How
>>>>>>>>does
>>>>>>>>that fit in?
>>>>>>>>
>>>>>>>>Best regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> - Andy
>>>>>>>>
>>>>>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>>>Hein
>>>>>>>>(via Tom White)
>>>>>>>>
>>>>>>>>
>>>>>>>>>________________________________
>>>>>>>>>From: Todd Lipcon <to...@cloudera.com>
>>>>>>>>>To: dev@hbase.apache.org
>>>>>>>>>Sent: Friday, July 29, 2011 9:58 AM
>>>>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>>>>
>>>>>>>>>What I'd prefer is something like:
>>>>>>>>>
>>>>>>>>>interface BlockCacheReportSink {
>>>>>>>>> public void reportStats(BlockCacheReport report);
>>>>>>>>>}
>>>>>>>>>
>>>>>>>>>class LoggingBlockCacheReportSink {
>>>>>>>>> ... {
>>>>>>>>> log it with whatever formatting you want
>>>>>>>>> }
>>>>>>>>>}
>>>>>>>>>
>>>>>>>>>then a configuration which could default to the logging
>>>>>>>>>implementation,
>>>>>>>>>but
>>>>>>>>>orgs could easily substitute their own implementation. For example,
>>>>>>>>>I
>>>>>>>>>could
>>>>>>>>>see wanting to do an implementation where it keeps local RRD graphs
>>>>>>>>>of
>>>>>>>>>some
>>>>>>>>>stats, or pushes them to a central management server.
>>>>>>>>>
>>>>>>>>>The assumption is that BlockCacheReport is a fairly straightforward
>>>>>>>>>"struct"
>>>>>>>>>with the non-formatted information available.
>>>>>>>>>
>>>>>>>>>-Todd
>>>>>>>>>
>>>>>>>>>On Fri, Jul 29, 2011 at 4:15 AM, Doug Meil
>>>>>>>>><do...@explorysmedical.com>wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Folks-
>>>>>>>>>>
>>>>>>>>>> You probably already my email yesterday on this...
>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-4089 (block cache
>>>>>>>>>>report)
>>>>>>>>>>
>>>>>>>>>> ...and I just created this one...
>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-4147 (StoreFile
>>>>>>>>>>query
>>>>>>>>>> report)
>>>>>>>>>>
>>>>>>>>>> What I'd like to run past the dev-list is this: if Hbase had
>>>>>>>>>>periodic
>>>>>>>>>> summary usage statistics, where should they go? What I'd like to
>>>>>>>>>>throw
>>>>>>>>>> out for discussion is that I'm suggesting that it should simply
>>>>>>>>>>go
>>>>>>>>>>to
>>>>>>>>>>the
>>>>>>>>>> log files and users can slice and dice this on their own. No UI
>>>>>>>>>>(I.e.,
>>>>>>>>>> JSPs), no JMX, etc.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The summary out the output is this:
>>>>>>>>>> BlockCacheReport: on configured interval, print out summary of
>>>>>>>>>>blockcache
>>>>>>>>>> (at table/CF level) to log file. This one is point-in-time, not
>>>>>>>>>>delta.
>>>>>>>>>>
>>>>>>>>>> StoreFile read report: on configured interval, print out summary
>>>>>>>>>>of
>>>>>>>>>> StoreFile accesses and how much time was spent reading each
>>>>>>>>>>StoreFile
>>>>>>>>>>to
>>>>>>>>>> log file.
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>> Doug
>>>>>>>>>>
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>--
>>>>>>>>>Todd Lipcon
>>>>>>>>>Software Engineer, Cloudera
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>
>
>
>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Ryan Rawson <ry...@gmail.com>.
Cassandra is not a big table clone.
Bigtable specifies 2 things:
- a data model with column families, etc
- a distribution model, a distributed architecture, etc
Only HBase provides both these things. Cassandra provides elements of
(a) and uses the dynamo model to do distribution.
I believe in HBase because I think it's distribution model is simpler,
more robust and reliable than the dynamo model.
On Mon, Aug 1, 2011 at 7:40 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>
>>Doug, sorry for off-topic, it was extremely hot day; and I finally found
>>that it is cheaper to get single dedicated cc1.4xlarge (EC2) for
>>$1.60/hour than to rely on advocated (Whirr and Friends) c1.xlarge
>>($1.68/hour); unbelievable (hot!) 10 minutes for garbage collection of 1
>>Gb heap only, 1000 (small!!!) transactions PER MINUTE only. It can be
>>explained by "virtual" sharing of the same cc1.4xlarge with at least two
>>of other users (3 x $1.68 = $2.04) . Especially "swap", "cache" is used
>>instead of file, and (of course) it is slower.
> No any problem with Hadoop thoughŠ
>
> BTW, I need to compare with Cassandra, another Big Table clone.
>
> (P.S. I used dedicated servers before, and never had problems with Hbase;
> except EC2)
>
> -Fuad
>
>
>
Re: Msft.... (renamed thread)
Posted by Ryan Rawson <ry...@gmail.com>.
Stack can talk about this, but essentially for a period he could
contribute to HBase only, but not to Hadoop. As you note, since Stack
has joined my former employer the situation is good now.
While it might be technically correct to say that MSFT has supported
HBase in the past, this is a legalistic view imo, since ultimately I
dont think that powerset uses HBase anymore.
-ryan
On Tue, Aug 2, 2011 at 8:59 AM, Andrew Purtell <ap...@apache.org> wrote:
> When Microsoft acquired Powerset, Stack and Jim were still working there, but were disallowed by policy to contribute to HBase ... for months. My understanding is this was due to concerns about intellectual property -- open source fright?. Anyway, it was a disruptive period for the project that was resolved when Stack left.
>
> Best regards,
>
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>>________________________________
>>From: Doug Meil <do...@explorysmedical.com>
>>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>>Sent: Tuesday, August 2, 2011 4:43 AM
>>Subject: Msft.... (renamed thread)
>>
>>
>>This is a reasonably interesting history question. Powerset folks were
>>working on Hbase in 2007, but per...
>>
>>http://en.wikipedia.org/wiki/Powerset_%28company%29
>>
>>... Microsoft didn't buy Powerset until mid-2008. But that's all in the
>>past.
>>
>>
>>However, Microsoft is currently a platinum sponsor of the Apache Software
>>Foundation...
>>
>>http://www.apache.org/foundation/thanks.html
>>
>>... along with Google and Yahoo, which means they all each donate at least
>>$100k per year to ASF.
>>
>>
>>So in an extended financial sense, Msft "supports" Hbase by way of their
>>donation to ASF, but they also support everything else in ASF, just like
>>the rest of the big donors.
>>
>>
>>
>>On 8/1/11 10:56 PM, "Ryan Rawson" <ry...@gmail.com> wrote:
>>
>>>No one at powerset is currently contributing to HBase. "Was" is the
>>>key here - in the past, etc.
>>>
>>>I guess MSFT never got to integrating the HBase API with the C# LINQ
>>>system and Visual Studio. Maybe it's that azure table services?
>>>
>>>
>>>
>>>On Mon, Aug 1, 2011 at 7:50 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>>>>
>>>>>
>>>>>
>>>>>re: "Is it really-really supported by Microsoft employees?!"
>>>>>
>>>>>It is really, really not.
>>>>>
>>>>
>>>>
>>>> I believe Hbase was contributed to Apache by a Powerset, currently owned
>>>> by Microsoft; and (same contributors) were full-time supporting Hbase
>>>>and
>>>> having salaries from Microsoft for at least a year; it was first
>>>> (implicit) contribution from Microsoft to Apache.
>>>>
>>>>
>>>>
>>
>>
>>
>>
Re: Msft.... (renamed thread)
Posted by Andrew Purtell <ap...@apache.org>.
When Microsoft acquired Powerset, Stack and Jim were still working there, but were disallowed by policy to contribute to HBase ... for months. My understanding is this was due to concerns about intellectual property -- open source fright?. Anyway, it was a disruptive period for the project that was resolved when Stack left.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Doug Meil <do...@explorysmedical.com>
>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>Sent: Tuesday, August 2, 2011 4:43 AM
>Subject: Msft.... (renamed thread)
>
>
>This is a reasonably interesting history question. Powerset folks were
>working on Hbase in 2007, but per...
>
>http://en.wikipedia.org/wiki/Powerset_%28company%29
>
>... Microsoft didn't buy Powerset until mid-2008. But that's all in the
>past.
>
>
>However, Microsoft is currently a platinum sponsor of the Apache Software
>Foundation...
>
>http://www.apache.org/foundation/thanks.html
>
>... along with Google and Yahoo, which means they all each donate at least
>$100k per year to ASF.
>
>
>So in an extended financial sense, Msft "supports" Hbase by way of their
>donation to ASF, but they also support everything else in ASF, just like
>the rest of the big donors.
>
>
>
>On 8/1/11 10:56 PM, "Ryan Rawson" <ry...@gmail.com> wrote:
>
>>No one at powerset is currently contributing to HBase. "Was" is the
>>key here - in the past, etc.
>>
>>I guess MSFT never got to integrating the HBase API with the C# LINQ
>>system and Visual Studio. Maybe it's that azure table services?
>>
>>
>>
>>On Mon, Aug 1, 2011 at 7:50 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>>>
>>>>
>>>>
>>>>re: "Is it really-really supported by Microsoft employees?!"
>>>>
>>>>It is really, really not.
>>>>
>>>
>>>
>>> I believe Hbase was contributed to Apache by a Powerset, currently owned
>>> by Microsoft; and (same contributors) were full-time supporting Hbase
>>>and
>>> having salaries from Microsoft for at least a year; it was first
>>> (implicit) contribution from Microsoft to Apache.
>>>
>>>
>>>
>
>
>
>
Msft.... (renamed thread)
Posted by Doug Meil <do...@explorysmedical.com>.
This is a reasonably interesting history question. Powerset folks were
working on Hbase in 2007, but per...
http://en.wikipedia.org/wiki/Powerset_%28company%29
... Microsoft didn't buy Powerset until mid-2008. But that's all in the
past.
However, Microsoft is currently a platinum sponsor of the Apache Software
Foundation...
http://www.apache.org/foundation/thanks.html
... along with Google and Yahoo, which means they all each donate at least
$100k per year to ASF.
So in an extended financial sense, Msft "supports" Hbase by way of their
donation to ASF, but they also support everything else in ASF, just like
the rest of the big donors.
On 8/1/11 10:56 PM, "Ryan Rawson" <ry...@gmail.com> wrote:
>No one at powerset is currently contributing to HBase. "Was" is the
>key here - in the past, etc.
>
>I guess MSFT never got to integrating the HBase API with the C# LINQ
>system and Visual Studio. Maybe it's that azure table services?
>
>
>
>On Mon, Aug 1, 2011 at 7:50 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>>
>>>
>>>
>>>re: "Is it really-really supported by Microsoft employees?!"
>>>
>>>It is really, really not.
>>>
>>
>>
>> I believe Hbase was contributed to Apache by a Powerset, currently owned
>> by Microsoft; and (same contributors) were full-time supporting Hbase
>>and
>> having salaries from Microsoft for at least a year; it was first
>> (implicit) contribution from Microsoft to Apache.
>>
>>
>>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Ryan Rawson <ry...@gmail.com>.
No one at powerset is currently contributing to HBase. "Was" is the
key here - in the past, etc.
I guess MSFT never got to integrating the HBase API with the C# LINQ
system and Visual Studio. Maybe it's that azure table services?
On Mon, Aug 1, 2011 at 7:50 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>
>>
>>
>>re: "Is it really-really supported by Microsoft employees?!"
>>
>>It is really, really not.
>>
>
>
> I believe Hbase was contributed to Apache by a Powerset, currently owned
> by Microsoft; and (same contributors) were full-time supporting Hbase and
> having salaries from Microsoft for at least a year; it was first
> (implicit) contribution from Microsoft to Apache.
>
>
>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Fuad Efendi <fu...@efendi.ca>.
>
>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Fuad Efendi <fu...@efendi.ca>.
>
>Doug, sorry for off-topic, it was extremely hot day; and I finally found
>that it is cheaper to get single dedicated cc1.4xlarge (EC2) for
>$1.60/hour than to rely on advocated (Whirr and Friends) c1.xlarge
>($1.68/hour); unbelievable (hot!) 10 minutes for garbage collection of 1
>Gb heap only, 1000 (small!!!) transactions PER MINUTE only. It can be
>explained by "virtual" sharing of the same cc1.4xlarge with at least two
>of other users (3 x $1.68 = $2.04) . Especially "swap", "cache" is used
>instead of file, and (of course) it is slower.
No any problem with Hadoop though
BTW, I need to compare with Cassandra, another Big Table clone.
(P.S. I used dedicated servers before, and never had problems with Hbase;
except EC2)
-Fuad
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Stack <sa...@gmail.com>.
Yes. Hbase was started by powerset and two of us contributed first as powerset employees and then later after our acquisition by microsoft and a pause of a few months while some interesting legal issues were worked through, i resumed contributing to apache hbase as an employee of Microsoft. A bunch of backscenes work by mgmnt at powerset and by the ms integration team made what I thought the impossible happen (at the time I was highly skeptical and was out talking to all and sundry looking for someone else to assume hbase sponsorship). I do not think our work was the first open source ms contrib -- I seem to remember fixes for some sqlserver open source driver happening before we were let resume -- but I am pretty sure it was the most sustained (I was near full time working on hbase). I remained a ms employee free to contrib to hbase, hadoop itself had not been vetted by ms legal so was off limits, for over a year until the day we all realized hbase no longer fit the direction in which the powerset division was headed. Then I moved on.
Microsoft sponsored early hbase dev.
Stack
On Aug 1, 2011, at 19:50, Fuad Efendi <fu...@efendi.ca> wrote:
>>
>>
>>
>> re: "Is it really-really supported by Microsoft employees?!"
>>
>> It is really, really not.
>>
>
>
> I believe Hbase was contributed to Apache by a Powerset, currently owned
> by Microsoft; and (same contributors) were full-time supporting Hbase and
> having salaries from Microsoft for at least a year; it was first
> (implicit) contribution from Microsoft to Apache.
>
>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Fuad Efendi <fu...@efendi.ca>.
>
>
>
>re: "Is it really-really supported by Microsoft employees?!"
>
>It is really, really not.
>
I believe Hbase was contributed to Apache by a Powerset, currently owned
by Microsoft; and (same contributors) were full-time supporting Hbase and
having salaries from Microsoft for at least a year; it was first
(implicit) contribution from Microsoft to Apache.
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Doug Meil <do...@explorysmedical.com>.
re: "Whirr"
I'm sure Cloudera would love to hear your feedback on Whirr, but please
address it to them directly and not on the Hbase dist-list.
re: "Is it really-really supported by Microsoft employees?!"
It is really, really not.
As you pointed out, and as is cited in the Apache Hbase book, Hbase was
inspired from BigTable. And as for the refund that Ryan suggested I'm
sure Google would be happy to make good on it. Watch out for the exchange
rate at the border.
On 7/31/11 6:38 PM, "Fuad Efendi" <fu...@efendi.ca> wrote:
>Great,
>
>
>
>My VP is former MS-98 :)))
>
>
>
>(BTW, many thanks to http://www.cloudera.com/ employees; please make Whirr
>really "rrrrrŠŠŠ" "open source"? I don't see any meaningŠ reinventing a
>bike is cheaper nowadays!!!)
>
>
>
>
>
>
>And, SEO of course: (remember: Google is SMARTER!!! You are just "clone".)
>
>==========================================================================
>
>
>--
>Fuad Efendi
>416-993-2060
>Tokenizer Inc., Canada
>Data Mining, Search Engines
>http://www.tokenizer.ca
>
>
>
>
>
>
>
>
>On 11-07-31 6:21 PM, "Ryan Rawson" <ry...@gmail.com> wrote:
>
>>You should ask for your money back!!
>>
>>On Sun, Jul 31, 2011 at 3:10 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>>> What is it all about? HBase sucks. Too many problems to newcomers,
>>> few-weeks-warm-up to begin with!!!!!!!!!!!!!!!! Is it really-really
>>> supported by Microsoft employees?!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> And, SEO of course:
>>> ===================
>>>
>>>
>>> --
>>> Fuad Efendi
>>> 416-993-2060
>>> Tokenizer Inc., Canada
>>> Data Mining, Search Engines
>>> http://www.tokenizer.ca
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 11-07-29 7:49 PM, "Otis Gospodnetic" <ot...@yahoo.com>
>>>wrote:
>>>
>>>>Hi,
>>>>
>>>>I'm for publishing all performance metrics in JMX (in addition to
>>>>exposing it wherever else you guys decide). That's because JMX is
>>>>probably the easiest for our SPM for HBase [1] to get to HBase
>>>>performance metrics and I suspect we are not alone.
>>>>
>>>>Otis
>>>>[1] http://sematext.com/spm/hbase-performance-monitoring/index.html
>>>>----
>>>>Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
>>>>Hadoop ecosystem search :: http://search-hadoop.com/
>>>>
>>>>
>>>>
>>>>>________________________________
>>>>>From: Andrew Purtell <ap...@apache.org>
>>>>>To: Doug Meil <do...@explorysmedical.com>; "dev@hbase.apache.org"
>>>>><de...@hbase.apache.org>
>>>>>Sent: Friday, July 29, 2011 4:34 PM
>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>
>>>>>> I'd rather see this output being able to be captured by something
>>>>>>the
>>>>>>sink that Todd suggested, rather than focusing on shell access.
>>>>>
>>>>>
>>>>>I don't agree.
>>>>>
>>>>>
>>>>>Look at what we have existing and proposed:
>>>>>
>>>>> - Java API access to server and region load information, that the
>>>>>shell uses
>>>>>
>>>>> - A proposal to dump some stats into log files, that then has to
>>>>>be
>>>>>scraped
>>>>>
>>>>> - A proposal (by the FB guys) to export some JSON via a HTTP
>>>>>servlet
>>>>>
>>>>>This is not good design, this is a bunch of random shit stuck
>>>>>together.
>>>>>
>>>>>Note that what Todd proposed does not preclude adding Java client API
>>>>>support for retrieving it.
>>>>>
>>>>>At a minimum all of this information must be accessible via the Java
>>>>>client API, to enable programmatic monitoring and analysis use cases.
>>>>>I'll add the shell support if nobody else cares about it, that is a
>>>>>relatively small detail, but one I think is important.
>>>>>
>>>>>Best regards,
>>>>>
>>>>>
>>>>> - Andy
>>>>>
>>>>>
>>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>Hein
>>>>>(via Tom White)
>>>>>
>>>>>
>>>>>>________________________________
>>>>>>From: Doug Meil <do...@explorysmedical.com>
>>>>>>To: "dev@hbase.apache.org" <de...@hbase.apache.org>;
>>>>>>"apurtell@apache.org" <ap...@apache.org>
>>>>>>Sent: Friday, July 29, 2011 11:39 AM
>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>
>>>>>>
>>>>>>I'd rather see this output being able to be captured by something the
>>>>>>sink
>>>>>>that Todd suggested, rather than focusing on shell access.
>>>>>>HServerLoad
>>>>>>is
>>>>>>super-summary at the RS level, and both the items in 4089 and 4147
>>>>>>are
>>>>>>proposed to be "summarized" but still have reasonable detail (e.g.,
>>>>>>even
>>>>>>table/CF summary there could be dozens of entries given a reasonably
>>>>>>complex system).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>On 7/29/11 1:15 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>>>>>>
>>>>>>>There is also the matter of HServerLoad and how that is used by the
>>>>>>>shell
>>>>>>>and master UI to report on cluster status.
>>>>>>>
>>>>>>>I'd like the shell to be able to let the user explore all of these
>>>>>>>different reports interactively.
>>>>>>>
>>>>>>>At the very least, they should all be handled the same way.
>>>>>>>
>>>>>>>And then there is Riley's work over at FB on a slow query log. How
>>>>>>>does
>>>>>>>that fit in?
>>>>>>>
>>>>>>>Best regards,
>>>>>>>
>>>>>>>
>>>>>>> - Andy
>>>>>>>
>>>>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>>Hein
>>>>>>>(via Tom White)
>>>>>>>
>>>>>>>
>>>>>>>>________________________________
>>>>>>>>From: Todd Lipcon <to...@cloudera.com>
>>>>>>>>To: dev@hbase.apache.org
>>>>>>>>Sent: Friday, July 29, 2011 9:58 AM
>>>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>>>
>>>>>>>>What I'd prefer is something like:
>>>>>>>>
>>>>>>>>interface BlockCacheReportSink {
>>>>>>>> public void reportStats(BlockCacheReport report);
>>>>>>>>}
>>>>>>>>
>>>>>>>>class LoggingBlockCacheReportSink {
>>>>>>>> ... {
>>>>>>>> log it with whatever formatting you want
>>>>>>>> }
>>>>>>>>}
>>>>>>>>
>>>>>>>>then a configuration which could default to the logging
>>>>>>>>implementation,
>>>>>>>>but
>>>>>>>>orgs could easily substitute their own implementation. For example,
>>>>>>>>I
>>>>>>>>could
>>>>>>>>see wanting to do an implementation where it keeps local RRD graphs
>>>>>>>>of
>>>>>>>>some
>>>>>>>>stats, or pushes them to a central management server.
>>>>>>>>
>>>>>>>>The assumption is that BlockCacheReport is a fairly straightforward
>>>>>>>>"struct"
>>>>>>>>with the non-formatted information available.
>>>>>>>>
>>>>>>>>-Todd
>>>>>>>>
>>>>>>>>On Fri, Jul 29, 2011 at 4:15 AM, Doug Meil
>>>>>>>><do...@explorysmedical.com>wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Folks-
>>>>>>>>>
>>>>>>>>> You probably already my email yesterday on this...
>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-4089 (block cache
>>>>>>>>>report)
>>>>>>>>>
>>>>>>>>> ...and I just created this one...
>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-4147 (StoreFile
>>>>>>>>>query
>>>>>>>>> report)
>>>>>>>>>
>>>>>>>>> What I'd like to run past the dev-list is this: if Hbase had
>>>>>>>>>periodic
>>>>>>>>> summary usage statistics, where should they go? What I'd like to
>>>>>>>>>throw
>>>>>>>>> out for discussion is that I'm suggesting that it should simply
>>>>>>>>>go
>>>>>>>>>to
>>>>>>>>>the
>>>>>>>>> log files and users can slice and dice this on their own. No UI
>>>>>>>>>(I.e.,
>>>>>>>>> JSPs), no JMX, etc.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The summary out the output is this:
>>>>>>>>> BlockCacheReport: on configured interval, print out summary of
>>>>>>>>>blockcache
>>>>>>>>> (at table/CF level) to log file. This one is point-in-time, not
>>>>>>>>>delta.
>>>>>>>>>
>>>>>>>>> StoreFile read report: on configured interval, print out summary
>>>>>>>>>of
>>>>>>>>> StoreFile accesses and how much time was spent reading each
>>>>>>>>>StoreFile
>>>>>>>>>to
>>>>>>>>> log file.
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> Doug
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>--
>>>>>>>>Todd Lipcon
>>>>>>>>Software Engineer, Cloudera
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>>
>
>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Fuad Efendi <fu...@efendi.ca>.
Great,
My VP is former MS-98 :)))
(BTW, many thanks to http://www.cloudera.com/ employees; please make Whirr
really "rrrrr" "open source"? I don't see any meaning reinventing a
bike is cheaper nowadays!!!)
And, SEO of course: (remember: Google is SMARTER!!! You are just "clone".)
==========================================================================
--
Fuad Efendi
416-993-2060
Tokenizer Inc., Canada
Data Mining, Search Engines
http://www.tokenizer.ca
On 11-07-31 6:21 PM, "Ryan Rawson" <ry...@gmail.com> wrote:
>You should ask for your money back!!
>
>On Sun, Jul 31, 2011 at 3:10 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>> What is it all about? HBase sucks. Too many problems to newcomers,
>> few-weeks-warm-up to begin with!!!!!!!!!!!!!!!! Is it really-really
>> supported by Microsoft employees?!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> And, SEO of course:
>> ===================
>>
>>
>> --
>> Fuad Efendi
>> 416-993-2060
>> Tokenizer Inc., Canada
>> Data Mining, Search Engines
>> http://www.tokenizer.ca
>>
>>
>>
>>
>>
>>
>>
>>
>> On 11-07-29 7:49 PM, "Otis Gospodnetic" <ot...@yahoo.com>
>>wrote:
>>
>>>Hi,
>>>
>>>I'm for publishing all performance metrics in JMX (in addition to
>>>exposing it wherever else you guys decide). That's because JMX is
>>>probably the easiest for our SPM for HBase [1] to get to HBase
>>>performance metrics and I suspect we are not alone.
>>>
>>>Otis
>>>[1] http://sematext.com/spm/hbase-performance-monitoring/index.html
>>>----
>>>Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
>>>Hadoop ecosystem search :: http://search-hadoop.com/
>>>
>>>
>>>
>>>>________________________________
>>>>From: Andrew Purtell <ap...@apache.org>
>>>>To: Doug Meil <do...@explorysmedical.com>; "dev@hbase.apache.org"
>>>><de...@hbase.apache.org>
>>>>Sent: Friday, July 29, 2011 4:34 PM
>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>
>>>>> I'd rather see this output being able to be captured by something the
>>>>>sink that Todd suggested, rather than focusing on shell access.
>>>>
>>>>
>>>>I don't agree.
>>>>
>>>>
>>>>Look at what we have existing and proposed:
>>>>
>>>> - Java API access to server and region load information, that the
>>>>shell uses
>>>>
>>>> - A proposal to dump some stats into log files, that then has to be
>>>>scraped
>>>>
>>>> - A proposal (by the FB guys) to export some JSON via a HTTP
>>>>servlet
>>>>
>>>>This is not good design, this is a bunch of random shit stuck together.
>>>>
>>>>Note that what Todd proposed does not preclude adding Java client API
>>>>support for retrieving it.
>>>>
>>>>At a minimum all of this information must be accessible via the Java
>>>>client API, to enable programmatic monitoring and analysis use cases.
>>>>I'll add the shell support if nobody else cares about it, that is a
>>>>relatively small detail, but one I think is important.
>>>>
>>>>Best regards,
>>>>
>>>>
>>>> - Andy
>>>>
>>>>
>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>Hein
>>>>(via Tom White)
>>>>
>>>>
>>>>>________________________________
>>>>>From: Doug Meil <do...@explorysmedical.com>
>>>>>To: "dev@hbase.apache.org" <de...@hbase.apache.org>;
>>>>>"apurtell@apache.org" <ap...@apache.org>
>>>>>Sent: Friday, July 29, 2011 11:39 AM
>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>
>>>>>
>>>>>I'd rather see this output being able to be captured by something the
>>>>>sink
>>>>>that Todd suggested, rather than focusing on shell access.
>>>>>HServerLoad
>>>>>is
>>>>>super-summary at the RS level, and both the items in 4089 and 4147 are
>>>>>proposed to be "summarized" but still have reasonable detail (e.g.,
>>>>>even
>>>>>table/CF summary there could be dozens of entries given a reasonably
>>>>>complex system).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>On 7/29/11 1:15 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>>>>>
>>>>>>There is also the matter of HServerLoad and how that is used by the
>>>>>>shell
>>>>>>and master UI to report on cluster status.
>>>>>>
>>>>>>I'd like the shell to be able to let the user explore all of these
>>>>>>different reports interactively.
>>>>>>
>>>>>>At the very least, they should all be handled the same way.
>>>>>>
>>>>>>And then there is Riley's work over at FB on a slow query log. How
>>>>>>does
>>>>>>that fit in?
>>>>>>
>>>>>>Best regards,
>>>>>>
>>>>>>
>>>>>> - Andy
>>>>>>
>>>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>Hein
>>>>>>(via Tom White)
>>>>>>
>>>>>>
>>>>>>>________________________________
>>>>>>>From: Todd Lipcon <to...@cloudera.com>
>>>>>>>To: dev@hbase.apache.org
>>>>>>>Sent: Friday, July 29, 2011 9:58 AM
>>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>>
>>>>>>>What I'd prefer is something like:
>>>>>>>
>>>>>>>interface BlockCacheReportSink {
>>>>>>> public void reportStats(BlockCacheReport report);
>>>>>>>}
>>>>>>>
>>>>>>>class LoggingBlockCacheReportSink {
>>>>>>> ... {
>>>>>>> log it with whatever formatting you want
>>>>>>> }
>>>>>>>}
>>>>>>>
>>>>>>>then a configuration which could default to the logging
>>>>>>>implementation,
>>>>>>>but
>>>>>>>orgs could easily substitute their own implementation. For example,
>>>>>>>I
>>>>>>>could
>>>>>>>see wanting to do an implementation where it keeps local RRD graphs
>>>>>>>of
>>>>>>>some
>>>>>>>stats, or pushes them to a central management server.
>>>>>>>
>>>>>>>The assumption is that BlockCacheReport is a fairly straightforward
>>>>>>>"struct"
>>>>>>>with the non-formatted information available.
>>>>>>>
>>>>>>>-Todd
>>>>>>>
>>>>>>>On Fri, Jul 29, 2011 at 4:15 AM, Doug Meil
>>>>>>><do...@explorysmedical.com>wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Folks-
>>>>>>>>
>>>>>>>> You probably already my email yesterday on this...
>>>>>>>> https://issues.apache.org/jira/browse/HBASE-4089 (block cache
>>>>>>>>report)
>>>>>>>>
>>>>>>>> ...and I just created this one...
>>>>>>>> https://issues.apache.org/jira/browse/HBASE-4147 (StoreFile query
>>>>>>>> report)
>>>>>>>>
>>>>>>>> What I'd like to run past the dev-list is this: if Hbase had
>>>>>>>>periodic
>>>>>>>> summary usage statistics, where should they go? What I'd like to
>>>>>>>>throw
>>>>>>>> out for discussion is that I'm suggesting that it should simply go
>>>>>>>>to
>>>>>>>>the
>>>>>>>> log files and users can slice and dice this on their own. No UI
>>>>>>>>(I.e.,
>>>>>>>> JSPs), no JMX, etc.
>>>>>>>>
>>>>>>>>
>>>>>>>> The summary out the output is this:
>>>>>>>> BlockCacheReport: on configured interval, print out summary of
>>>>>>>>blockcache
>>>>>>>> (at table/CF level) to log file. This one is point-in-time, not
>>>>>>>>delta.
>>>>>>>>
>>>>>>>> StoreFile read report: on configured interval, print out summary
>>>>>>>>of
>>>>>>>> StoreFile accesses and how much time was spent reading each
>>>>>>>>StoreFile
>>>>>>>>to
>>>>>>>> log file.
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> Doug
>>>>>>>>
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>--
>>>>>>>Todd Lipcon
>>>>>>>Software Engineer, Cloudera
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>>
Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
Posted by Ryan Rawson <ry...@gmail.com>.
You should ask for your money back!!
On Sun, Jul 31, 2011 at 3:10 PM, Fuad Efendi <fu...@efendi.ca> wrote:
> What is it all about? HBase sucks. Too many problems to newcomers,
> few-weeks-warm-up to begin with!!!!!!!!!!!!!!!! Is it really-really
> supported by Microsoft employees?!
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> And, SEO of course:
> ===================
>
>
> --
> Fuad Efendi
> 416-993-2060
> Tokenizer Inc., Canada
> Data Mining, Search Engines
> http://www.tokenizer.ca
>
>
>
>
>
>
>
>
> On 11-07-29 7:49 PM, "Otis Gospodnetic" <ot...@yahoo.com> wrote:
>
>>Hi,
>>
>>I'm for publishing all performance metrics in JMX (in addition to
>>exposing it wherever else you guys decide). That's because JMX is
>>probably the easiest for our SPM for HBase [1] to get to HBase
>>performance metrics and I suspect we are not alone.
>>
>>Otis
>>[1] http://sematext.com/spm/hbase-performance-monitoring/index.html
>>----
>>Sematext :: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
>>Hadoop ecosystem search :: http://search-hadoop.com/
>>
>>
>>
>>>________________________________
>>>From: Andrew Purtell <ap...@apache.org>
>>>To: Doug Meil <do...@explorysmedical.com>; "dev@hbase.apache.org"
>>><de...@hbase.apache.org>
>>>Sent: Friday, July 29, 2011 4:34 PM
>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>
>>>> I'd rather see this output being able to be captured by something the
>>>>sink that Todd suggested, rather than focusing on shell access.
>>>
>>>
>>>I don't agree.
>>>
>>>
>>>Look at what we have existing and proposed:
>>>
>>> - Java API access to server and region load information, that the
>>>shell uses
>>>
>>> - A proposal to dump some stats into log files, that then has to be
>>>scraped
>>>
>>> - A proposal (by the FB guys) to export some JSON via a HTTP servlet
>>>
>>>This is not good design, this is a bunch of random shit stuck together.
>>>
>>>Note that what Todd proposed does not preclude adding Java client API
>>>support for retrieving it.
>>>
>>>At a minimum all of this information must be accessible via the Java
>>>client API, to enable programmatic monitoring and analysis use cases.
>>>I'll add the shell support if nobody else cares about it, that is a
>>>relatively small detail, but one I think is important.
>>>
>>>Best regards,
>>>
>>>
>>> - Andy
>>>
>>>
>>>Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>(via Tom White)
>>>
>>>
>>>>________________________________
>>>>From: Doug Meil <do...@explorysmedical.com>
>>>>To: "dev@hbase.apache.org" <de...@hbase.apache.org>;
>>>>"apurtell@apache.org" <ap...@apache.org>
>>>>Sent: Friday, July 29, 2011 11:39 AM
>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>
>>>>
>>>>I'd rather see this output being able to be captured by something the
>>>>sink
>>>>that Todd suggested, rather than focusing on shell access. HServerLoad
>>>>is
>>>>super-summary at the RS level, and both the items in 4089 and 4147 are
>>>>proposed to be "summarized" but still have reasonable detail (e.g., even
>>>>table/CF summary there could be dozens of entries given a reasonably
>>>>complex system).
>>>>
>>>>
>>>>
>>>>
>>>>On 7/29/11 1:15 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>>>>
>>>>>There is also the matter of HServerLoad and how that is used by the
>>>>>shell
>>>>>and master UI to report on cluster status.
>>>>>
>>>>>I'd like the shell to be able to let the user explore all of these
>>>>>different reports interactively.
>>>>>
>>>>>At the very least, they should all be handled the same way.
>>>>>
>>>>>And then there is Riley's work over at FB on a slow query log. How does
>>>>>that fit in?
>>>>>
>>>>>Best regards,
>>>>>
>>>>>
>>>>> - Andy
>>>>>
>>>>>Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>Hein
>>>>>(via Tom White)
>>>>>
>>>>>
>>>>>>________________________________
>>>>>>From: Todd Lipcon <to...@cloudera.com>
>>>>>>To: dev@hbase.apache.org
>>>>>>Sent: Friday, July 29, 2011 9:58 AM
>>>>>>Subject: Re: HBASE-4089 & HBASE-4147 - on the topic of ops output
>>>>>>
>>>>>>What I'd prefer is something like:
>>>>>>
>>>>>>interface BlockCacheReportSink {
>>>>>> public void reportStats(BlockCacheReport report);
>>>>>>}
>>>>>>
>>>>>>class LoggingBlockCacheReportSink {
>>>>>> ... {
>>>>>> log it with whatever formatting you want
>>>>>> }
>>>>>>}
>>>>>>
>>>>>>then a configuration which could default to the logging
>>>>>>implementation,
>>>>>>but
>>>>>>orgs could easily substitute their own implementation. For example, I
>>>>>>could
>>>>>>see wanting to do an implementation where it keeps local RRD graphs of
>>>>>>some
>>>>>>stats, or pushes them to a central management server.
>>>>>>
>>>>>>The assumption is that BlockCacheReport is a fairly straightforward
>>>>>>"struct"
>>>>>>with the non-formatted information available.
>>>>>>
>>>>>>-Todd
>>>>>>
>>>>>>On Fri, Jul 29, 2011 at 4:15 AM, Doug Meil
>>>>>><do...@explorysmedical.com>wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi Folks-
>>>>>>>
>>>>>>> You probably already my email yesterday on this...
>>>>>>> https://issues.apache.org/jira/browse/HBASE-4089 (block cache
>>>>>>>report)
>>>>>>>
>>>>>>> ...and I just created this one...
>>>>>>> https://issues.apache.org/jira/browse/HBASE-4147 (StoreFile query
>>>>>>> report)
>>>>>>>
>>>>>>> What I'd like to run past the dev-list is this: if Hbase had
>>>>>>>periodic
>>>>>>> summary usage statistics, where should they go? What I'd like to
>>>>>>>throw
>>>>>>> out for discussion is that I'm suggesting that it should simply go
>>>>>>>to
>>>>>>>the
>>>>>>> log files and users can slice and dice this on their own. No UI
>>>>>>>(I.e.,
>>>>>>> JSPs), no JMX, etc.
>>>>>>>
>>>>>>>
>>>>>>> The summary out the output is this:
>>>>>>> BlockCacheReport: on configured interval, print out summary of
>>>>>>>blockcache
>>>>>>> (at table/CF level) to log file. This one is point-in-time, not
>>>>>>>delta.
>>>>>>>
>>>>>>> StoreFile read report: on configured interval, print out summary of
>>>>>>> StoreFile accesses and how much time was spent reading each
>>>>>>>StoreFile
>>>>>>>to
>>>>>>> log file.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Doug
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>--
>>>>>>Todd Lipcon
>>>>>>Software Engineer, Cloudera
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>
>
>