You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Joseph Coleman <jo...@infinitecampus.com> on 2011/07/25 17:09:01 UTC
Monitoring
Greetings,
I am relatively new to Hadoop but we now have an 10 node cluster up and running just DFS for now and will be expanding this rapidly as well as adding Hbase. I am looking to find out what people are using for monitoring Hadoop currently. I want to be notified if a node fails, performance statistics, failed drive or services ect. I was thinking of using Opsview and trying in Ganglia. Thanks in advance
Joe
Re: Monitoring
Posted by Joey Echeverria <jo...@cloudera.com>.
Hey Joe,
Hadoop and HBase are pretty monitoring tool agnostic. It does provide
a number of metrics via JMX and a REST interface which you can tie
into the monitoring tool of your choice. You can enable collection via
the REST service by editing
$HADOOP_HOME/conf/hadoop-metrics.properties and setting the *.class
settings, e.g.
# Configuration of the "dfs" context for /metrics
dfs.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
# Configuration of the "mapred" context for /metrics
mapred.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
Would configure both HDFS and MapReduce to make the metrics available,
but not write them to anything. There is also a GangliaContext for
integrating directly with Ganglia. Similar settings exist for HBase.
-Joey
On Mon, Jul 25, 2011 at 8:09 AM, Joseph Coleman
<jo...@infinitecampus.com> wrote:
> Greetings,
>
> I am relatively new to Hadoop but we now have an 10 node cluster up and running just DFS for now and will be expanding this rapidly as well as adding Hbase. I am looking to find out what people are using for monitoring Hadoop currently. I want to be notified if a node fails, performance statistics, failed drive or services ect. I was thinking of using Opsview and trying in Ganglia. Thanks in advance
>
> Joe
>
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: Monitoring
Posted by Stack <st...@duboce.net>.
I made a patch for the manual updating our Cloudera text some and
adding in MapR reference. I did it here:
https://issues.apache.org/jira/browse/HBASE-4140 My wordsmithing is
not the best so input appreciated. Will just commit tomorrow and push
it out if nought said.
St.Ack
On Mon, Jul 25, 2011 at 2:41 PM, Stack <st...@duboce.net> wrote:
> On Mon, Jul 25, 2011 at 1:20 PM, Buttler, David <bu...@llnl.gov> wrote:
>> However, only two vendors deliver a platform that supports hbase (with append): Cloudera and MapR. HortonWorks and ASF do not (to my knowledge). I am not sure I can count hard to find/compile branches that exist in ASF's version control as "supporting" hbase.
>>
>
> Yes.
>
> The manual says this on hadoop version currently:
>
> "This version of HBase will only run on Hadoop 0.20.x. It will not run
> on hadoop 0.21.x (nor 0.22.x). HBase will lose data unless it is
> running on an HDFS that has a durable sync. Hadoop 0.20.2 and Hadoop
> 0.20.203.0 DO NOT have this attribute. Currently only the
> branch-0.20-append branch has this....
>
> Or rather than build your own, you could use Cloudera's CDH3. CDH has
> the 0.20-append patches needed to add a durable sync (CDH3 betas will
> suffice; b2, b3, or b4)."
>
> Unless objection, I think I should add MapR to the tail of the last
> paragraph (with the 'free as in free beer' caveat).
>
> St.Ack
>
Re: Monitoring
Posted by Andrew Purtell <ap...@apache.org>.
> From: Jeff Whiting <je...@qualtrics.com>
> If Ted had answered the post along the lines of (I ganked the reply from Joey):
>
> Hadoop and HBase are pretty monitoring tool agnostic. It does provide
> a number of metrics via JMX and a REST interface which you can tie
> into the monitoring tool of your choice.
>
> Ted
>
> p.s. If you are using MapR it will do this for you out of the box.
>
> I don't think any of us would be having this conversion right now.
Thanks, Jeff, my thoughts exactly.
Instead we had a nearly content free plug for a commercial product on an FOSS user list. On the HBase lists to my knowledge this was the first, but I've been seeing such plugs for this particular commercial product frequently on various other Hadoop lists. Observing this is now equally predictable and annoying. I find such plugs to be nonresponsive to the concerns of FOSS users and a minor intrusion into our (FOSS) community.
- Andy
Re: Monitoring
Posted by Jeff Whiting <je...@qualtrics.com>.
It seems this is going to have to be something that is a judgment call. It will be hard to define
exactly when you should or shouldn't mention something. The general principle is that the ASF
option should always be touted first, followed by any other OSS / free options, then if no other
options exist the commercial software. I think Ted understood this when he started his email
"Slightly off topic..."
If Ted had answered the post along the lines of (I ganked the reply from Joey):
Hadoop and HBase are pretty monitoring tool agnostic. It does provide
a number of metrics via JMX and a REST interface which you can tie
into the monitoring tool of your choice.
Ted
p.s. If you are using MapR it will do this for you out of the box.
I don't think any of us would be having this conversion right now. The email would have promoted
the ASF, OSS option that is available to everyone. Also I wouldn't have a problem with the small
reference to MapR as it has a unique out of the box solution not available with the ASF release.
For the same reason I don't mind references to CDH3 for the hadoop append branch as it a solution
not found in an ASF release.
My 2 cents,
~Jeff
"If we have data we'll use data, if we have opinions we'll use mine."
On 7/25/2011 3:48 PM, Ryan Rawson wrote:
> I think it's fair to note which environments you can run HBase on top
> of. If we disallow that then we will have the tricky bit where there
> is no ASF release of Hadoop that is suitable to run HBase on top of.
> And who knows, perhaps the ceph guys, or openstack or<whatever> might
> come up with a suitable HDFS interop.
>
> In the mean time, what would be a good line to draw between acceptable
> vendor shilling, and unacceptable? The line seem fine, but also
> reasonably recognizable, eg: talking about why one might run HBase
> here or there seems ok, but talking about value add features in depth
> might not be. I just want HBase users to have a good experience
> running HBase, no matter where that might end up being.
>
> On Mon, Jul 25, 2011 at 2:41 PM, Stack<st...@duboce.net> wrote:
>> On Mon, Jul 25, 2011 at 1:20 PM, Buttler, David<bu...@llnl.gov> wrote:
>>> However, only two vendors deliver a platform that supports hbase (with append): Cloudera and MapR. HortonWorks and ASF do not (to my knowledge). I am not sure I can count hard to find/compile branches that exist in ASF's version control as "supporting" hbase.
>>>
>> Yes.
>>
>> The manual says this on hadoop version currently:
>>
>> "This version of HBase will only run on Hadoop 0.20.x. It will not run
>> on hadoop 0.21.x (nor 0.22.x). HBase will lose data unless it is
>> running on an HDFS that has a durable sync. Hadoop 0.20.2 and Hadoop
>> 0.20.203.0 DO NOT have this attribute. Currently only the
>> branch-0.20-append branch has this....
>>
>> Or rather than build your own, you could use Cloudera's CDH3. CDH has
>> the 0.20-append patches needed to add a durable sync (CDH3 betas will
>> suffice; b2, b3, or b4)."
>>
>> Unless objection, I think I should add MapR to the tail of the last
>> paragraph (with the 'free as in free beer' caveat).
>>
>> St.Ack
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: Monitoring
Posted by Ryan Rawson <ry...@gmail.com>.
I think it's fair to note which environments you can run HBase on top
of. If we disallow that then we will have the tricky bit where there
is no ASF release of Hadoop that is suitable to run HBase on top of.
And who knows, perhaps the ceph guys, or openstack or <whatever> might
come up with a suitable HDFS interop.
In the mean time, what would be a good line to draw between acceptable
vendor shilling, and unacceptable? The line seem fine, but also
reasonably recognizable, eg: talking about why one might run HBase
here or there seems ok, but talking about value add features in depth
might not be. I just want HBase users to have a good experience
running HBase, no matter where that might end up being.
On Mon, Jul 25, 2011 at 2:41 PM, Stack <st...@duboce.net> wrote:
> On Mon, Jul 25, 2011 at 1:20 PM, Buttler, David <bu...@llnl.gov> wrote:
>> However, only two vendors deliver a platform that supports hbase (with append): Cloudera and MapR. HortonWorks and ASF do not (to my knowledge). I am not sure I can count hard to find/compile branches that exist in ASF's version control as "supporting" hbase.
>>
>
> Yes.
>
> The manual says this on hadoop version currently:
>
> "This version of HBase will only run on Hadoop 0.20.x. It will not run
> on hadoop 0.21.x (nor 0.22.x). HBase will lose data unless it is
> running on an HDFS that has a durable sync. Hadoop 0.20.2 and Hadoop
> 0.20.203.0 DO NOT have this attribute. Currently only the
> branch-0.20-append branch has this....
>
> Or rather than build your own, you could use Cloudera's CDH3. CDH has
> the 0.20-append patches needed to add a durable sync (CDH3 betas will
> suffice; b2, b3, or b4)."
>
> Unless objection, I think I should add MapR to the tail of the last
> paragraph (with the 'free as in free beer' caveat).
>
> St.Ack
>
Re: Monitoring
Posted by Stack <st...@duboce.net>.
On Mon, Jul 25, 2011 at 1:20 PM, Buttler, David <bu...@llnl.gov> wrote:
> However, only two vendors deliver a platform that supports hbase (with append): Cloudera and MapR. HortonWorks and ASF do not (to my knowledge). I am not sure I can count hard to find/compile branches that exist in ASF's version control as "supporting" hbase.
>
Yes.
The manual says this on hadoop version currently:
"This version of HBase will only run on Hadoop 0.20.x. It will not run
on hadoop 0.21.x (nor 0.22.x). HBase will lose data unless it is
running on an HDFS that has a durable sync. Hadoop 0.20.2 and Hadoop
0.20.203.0 DO NOT have this attribute. Currently only the
branch-0.20-append branch has this....
Or rather than build your own, you could use Cloudera's CDH3. CDH has
the 0.20-append patches needed to add a durable sync (CDH3 betas will
suffice; b2, b3, or b4)."
Unless objection, I think I should add MapR to the tail of the last
paragraph (with the 'free as in free beer' caveat).
St.Ack
RE: Monitoring
Posted by "Buttler, David" <bu...@llnl.gov>.
However, only two vendors deliver a platform that supports hbase (with append): Cloudera and MapR. HortonWorks and ASF do not (to my knowledge). I am not sure I can count hard to find/compile branches that exist in ASF's version control as "supporting" hbase.
MapR and Cloudera both have free versions.
Cloudera has a license advantage.
I haven't finished a detailed feature comparison, so I can't comment there yet.
Dave
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Monday, July 25, 2011 1:10 PM
To: user@hbase.apache.org
Subject: Re: Monitoring
But surely for logical consistency, we should not favor one vendor (as
we have been for a year now), over another. So would it be correct to
continue to suggest to users they use CDH? After all, even though it
is ASF2.0 and free, it is still giving one vendor a leg up over others
(including hortonworks, the ASF project, etc).
On Mon, Jul 25, 2011 at 1:00 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Mon, Jul 25, 2011 at 11:55 AM, Ted Dunning <td...@maprtech.com> wrote:
>
>> Todd,
>>
>> Good to have you weigh in on this. You provide a good counterweight.
>>
>> To take a new hypothetical, suppose that one of the many, many patches that
>> Cloudera has championed for Hadoop is critical for Hbase operation or makes
>> Hbase faster.
>>
>> Is it reasonable to answer a question of the form "Is HDFS-xxx fixed?" with
>> "Fixed in CDH, followups off-list"?
>>
>
> One distinguishing factor is that CDH is Apache 2.0 licensed free and open
> source software. And the answer would never be "fixed in CDH but not in
> Apache trunk" -- any non-trivial changes in CDH are committed to trunk
> before we backport them. So, I feel like pointing people to CDH is
> appropriate for this open-source list.
>
> As for Cloudera Enterprise (our paid closed-source product) I'd expect to be
> held to the same standards as a MapR employee touting MapR -- i.e. I
> wouldn't bring it up on the public mailing list.
>
>
>> That seems to be important information for not just the original poster but
>> others who may have the same problem.
>>
>> What is the consensus on that?
>>
>> On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon <to...@cloudera.com> wrote:
>>
>> > > Another answer that I want to underscore is "MapR supports Hbase. A
>> > lot."
>> > >
>> >
>> > The issue is that I might be then tempted to start arguing that Cloudera
>> > Enterprise supports HBase better than MapR. Then we devolve into an
>> > annoying
>> > vendor war which doesn't help anyone.
>> >
>> > Best to just set a policy and stick to it for public responses. I don't
>> see
>> > anything wrong with your replying off-list to the requester with a sales
>> > pitch. This provides the information that might help the user without
>> > risking polluting the -user list with a lot of discussion of commercial
>> > products.
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Monitoring
Posted by Ted Dunning <td...@maprtech.com>.
On Mon, Jul 25, 2011 at 1:17 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Mon, Jul 25, 2011 at 1:09 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
> > But surely for logical consistency, we should not favor one vendor (as
> > we have been for a year now), over another. So would it be correct to
> > continue to suggest to users they use CDH? After all, even though it
> > is ASF2.0 and free, it is still giving one vendor a leg up over others
> > (including hortonworks, the ASF project, etc).
> >
>
> Most of the time that we suggest CDH, we also say that you could run the
> 0.20-append branch from the ASF. If Hortonworks had a free Apache 2.0
> licensed release that worked well with HBase we could recommend that, too.
> I
> wouldn't have a problem with any of the above.
>
My only test for these would be something that helps hbase users.
It definitely helps to know about Apache branches, CDH and MapR. As soon as
Apache and/or Hortonworks have a viable release for hbase, then they ought
to be on the list.
With a short list, I don't think order matters. If we get more than half a
dozen vendors (or orgs) then it might start to matter.
Re: Monitoring
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jul 25, 2011 at 1:09 PM, Ryan Rawson <ry...@gmail.com> wrote:
> But surely for logical consistency, we should not favor one vendor (as
> we have been for a year now), over another. So would it be correct to
> continue to suggest to users they use CDH? After all, even though it
> is ASF2.0 and free, it is still giving one vendor a leg up over others
> (including hortonworks, the ASF project, etc).
>
Most of the time that we suggest CDH, we also say that you could run the
0.20-append branch from the ASF. If Hortonworks had a free Apache 2.0
licensed release that worked well with HBase we could recommend that, too. I
wouldn't have a problem with any of the above.
-Todd
>
>
>
> On Mon, Jul 25, 2011 at 1:00 PM, Todd Lipcon <to...@cloudera.com> wrote:
> > On Mon, Jul 25, 2011 at 11:55 AM, Ted Dunning <td...@maprtech.com>
> wrote:
> >
> >> Todd,
> >>
> >> Good to have you weigh in on this. You provide a good counterweight.
> >>
> >> To take a new hypothetical, suppose that one of the many, many patches
> that
> >> Cloudera has championed for Hadoop is critical for Hbase operation or
> makes
> >> Hbase faster.
> >>
> >> Is it reasonable to answer a question of the form "Is HDFS-xxx fixed?"
> with
> >> "Fixed in CDH, followups off-list"?
> >>
> >
> > One distinguishing factor is that CDH is Apache 2.0 licensed free and
> open
> > source software. And the answer would never be "fixed in CDH but not in
> > Apache trunk" -- any non-trivial changes in CDH are committed to trunk
> > before we backport them. So, I feel like pointing people to CDH is
> > appropriate for this open-source list.
> >
> > As for Cloudera Enterprise (our paid closed-source product) I'd expect to
> be
> > held to the same standards as a MapR employee touting MapR -- i.e. I
> > wouldn't bring it up on the public mailing list.
> >
> >
> >> That seems to be important information for not just the original poster
> but
> >> others who may have the same problem.
> >>
> >> What is the consensus on that?
> >>
> >> On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon <to...@cloudera.com>
> wrote:
> >>
> >> > > Another answer that I want to underscore is "MapR supports Hbase. A
> >> > lot."
> >> > >
> >> >
> >> > The issue is that I might be then tempted to start arguing that
> Cloudera
> >> > Enterprise supports HBase better than MapR. Then we devolve into an
> >> > annoying
> >> > vendor war which doesn't help anyone.
> >> >
> >> > Best to just set a policy and stick to it for public responses. I
> don't
> >> see
> >> > anything wrong with your replying off-list to the requester with a
> sales
> >> > pitch. This provides the information that might help the user without
> >> > risking polluting the -user list with a lot of discussion of
> commercial
> >> > products.
> >> >
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Monitoring
Posted by Ryan Rawson <ry...@gmail.com>.
But surely for logical consistency, we should not favor one vendor (as
we have been for a year now), over another. So would it be correct to
continue to suggest to users they use CDH? After all, even though it
is ASF2.0 and free, it is still giving one vendor a leg up over others
(including hortonworks, the ASF project, etc).
On Mon, Jul 25, 2011 at 1:00 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Mon, Jul 25, 2011 at 11:55 AM, Ted Dunning <td...@maprtech.com> wrote:
>
>> Todd,
>>
>> Good to have you weigh in on this. You provide a good counterweight.
>>
>> To take a new hypothetical, suppose that one of the many, many patches that
>> Cloudera has championed for Hadoop is critical for Hbase operation or makes
>> Hbase faster.
>>
>> Is it reasonable to answer a question of the form "Is HDFS-xxx fixed?" with
>> "Fixed in CDH, followups off-list"?
>>
>
> One distinguishing factor is that CDH is Apache 2.0 licensed free and open
> source software. And the answer would never be "fixed in CDH but not in
> Apache trunk" -- any non-trivial changes in CDH are committed to trunk
> before we backport them. So, I feel like pointing people to CDH is
> appropriate for this open-source list.
>
> As for Cloudera Enterprise (our paid closed-source product) I'd expect to be
> held to the same standards as a MapR employee touting MapR -- i.e. I
> wouldn't bring it up on the public mailing list.
>
>
>> That seems to be important information for not just the original poster but
>> others who may have the same problem.
>>
>> What is the consensus on that?
>>
>> On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon <to...@cloudera.com> wrote:
>>
>> > > Another answer that I want to underscore is "MapR supports Hbase. A
>> > lot."
>> > >
>> >
>> > The issue is that I might be then tempted to start arguing that Cloudera
>> > Enterprise supports HBase better than MapR. Then we devolve into an
>> > annoying
>> > vendor war which doesn't help anyone.
>> >
>> > Best to just set a policy and stick to it for public responses. I don't
>> see
>> > anything wrong with your replying off-list to the requester with a sales
>> > pitch. This provides the information that might help the user without
>> > risking polluting the -user list with a lot of discussion of commercial
>> > products.
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Monitoring
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jul 25, 2011 at 11:55 AM, Ted Dunning <td...@maprtech.com> wrote:
> Todd,
>
> Good to have you weigh in on this. You provide a good counterweight.
>
> To take a new hypothetical, suppose that one of the many, many patches that
> Cloudera has championed for Hadoop is critical for Hbase operation or makes
> Hbase faster.
>
> Is it reasonable to answer a question of the form "Is HDFS-xxx fixed?" with
> "Fixed in CDH, followups off-list"?
>
One distinguishing factor is that CDH is Apache 2.0 licensed free and open
source software. And the answer would never be "fixed in CDH but not in
Apache trunk" -- any non-trivial changes in CDH are committed to trunk
before we backport them. So, I feel like pointing people to CDH is
appropriate for this open-source list.
As for Cloudera Enterprise (our paid closed-source product) I'd expect to be
held to the same standards as a MapR employee touting MapR -- i.e. I
wouldn't bring it up on the public mailing list.
> That seems to be important information for not just the original poster but
> others who may have the same problem.
>
> What is the consensus on that?
>
> On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > > Another answer that I want to underscore is "MapR supports Hbase. A
> > lot."
> > >
> >
> > The issue is that I might be then tempted to start arguing that Cloudera
> > Enterprise supports HBase better than MapR. Then we devolve into an
> > annoying
> > vendor war which doesn't help anyone.
> >
> > Best to just set a policy and stick to it for public responses. I don't
> see
> > anything wrong with your replying off-list to the requester with a sales
> > pitch. This provides the information that might help the user without
> > risking polluting the -user list with a lot of discussion of commercial
> > products.
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Monitoring
Posted by Jacob R Rideout <ap...@jacobrideout.net>.
> IMO, an answer that was just "Fixed in CDH, followups off-list" would
> be deserving of a yellow card.
>
> IMO, if the answer was 'No but it is fixed in CDH...', that might be
> sufficient (You've answered the question first and then diverted the
> user). If the 'No' and the '.. it is fixed...' clauses were further
> separated by say a slap for asking such a silly question on the list
> when the issue holds the answer or by adding some news on the state
> HDFS-xxx perhaps not yet in the issue, that'd make the mention of the
> commercial go down the easier.
>
On the IETF lists the convention is to distinguish between the cases
explicitly through meta-data in rather than assume the reader can
distinguish the authors intent or the organization basis/bias for
their statement. Example:
As an Apache Contributor:
HDFS-xxx is available in trunk and should address your issue.
As an employee of a commercial vendor:
Commercial product X includes patch HDFS-xxx. Please contact me
off-list if you want to learn more.
- Jacob Rideout
Re: Monitoring
Posted by Stack <st...@duboce.net>.
On Mon, Jul 25, 2011 at 11:55 AM, Ted Dunning <td...@maprtech.com> wrote:
> Is it reasonable to answer a question of the form "Is HDFS-xxx fixed?" with
> "Fixed in CDH, followups off-list"?
>
> That seems to be important information for not just the original poster but
> others who may have the same problem.
>
> What is the consensus on that?
>
IMO, an answer that was just "Fixed in CDH, followups off-list" would
be deserving of a yellow card.
IMO, if the answer was 'No but it is fixed in CDH...', that might be
sufficient (You've answered the question first and then diverted the
user). If the 'No' and the '.. it is fixed...' clauses were further
separated by say a slap for asking such a silly question on the list
when the issue holds the answer or by adding some news on the state
HDFS-xxx perhaps not yet in the issue, that'd make the mention of the
commercial go down the easier.
St.Ack
Re: Monitoring
Posted by Ted Dunning <td...@maprtech.com>.
Todd,
Good to have you weigh in on this. You provide a good counterweight.
To take a new hypothetical, suppose that one of the many, many patches that
Cloudera has championed for Hadoop is critical for Hbase operation or makes
Hbase faster.
Is it reasonable to answer a question of the form "Is HDFS-xxx fixed?" with
"Fixed in CDH, followups off-list"?
That seems to be important information for not just the original poster but
others who may have the same problem.
What is the consensus on that?
On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon <to...@cloudera.com> wrote:
> > Another answer that I want to underscore is "MapR supports Hbase. A
> lot."
> >
>
> The issue is that I might be then tempted to start arguing that Cloudera
> Enterprise supports HBase better than MapR. Then we devolve into an
> annoying
> vendor war which doesn't help anyone.
>
> Best to just set a policy and stick to it for public responses. I don't see
> anything wrong with your replying off-list to the requester with a sales
> pitch. This provides the information that might help the user without
> risking polluting the -user list with a lot of discussion of commercial
> products.
>
Re: Monitoring
Posted by Ted Dunning <td...@maprtech.com>.
Let's all resolve not to do that (on-list, particularly).
On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Then we devolve into an annoying
> vendor war which doesn't help anyone.
>
Re: Monitoring
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jul 25, 2011 at 6:10 PM, Doug Meil <do...@explorysmedical.com>wrote:
>
> Andrew: nicely put.
>
> Jeff W: I agree with your ordering.
>
> Stack: I agree with the book change for inclusion, and I agree with the
> caveat about 'free as in free beer'.
>
+1, seems entirely reasonable to list compatible alternatives in our docs.
>
> Todd/Ryan: regarding the book reference to CDH, I think the CDH reference
> being 'free as in beer' reference is an important differentiator, if it
> wasn't it would be a different story.
>
To nit-pick, CDH is free-as-in-speech as well as free-as-in-beer, since it's
under an Apache 2.0 license, same as HBase.
>
>
> Ted: re: "My only test for these would be something that helps hbase
> users", I agree with the intent, as long as the available license isn't
> only commercial.
>
>
>
>
> On 7/25/11 8:43 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>
> >I agree it's a fine line. As far as vendor specific pronouncements, may I
> >suggest a little goes a long way, quality over quantity.
> >
> >In my opinion it's never sufficient to say only "nonopen product Foo does
> >X" on an open source project's user list. Instead you need to first
> >explain how to do X with available FOSS tools. After that, mentioning
> >that Foo does it out of the box and is therefore easier than the
> >explained FOSS option is totally acceptable.
> >
> >Best regards,
> >
> >
> > - Andy
> >
> >Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >(via Tom White)
> >
> >
> >----- Original Message -----
> >> From: Todd Lipcon <to...@cloudera.com>
> >> To: user@hbase.apache.org
> >> Cc:
> >> Sent: Monday, July 25, 2011 11:45 AM
> >> Subject: Re: Monitoring
> >>
> >> On Mon, Jul 25, 2011 at 11:28 AM, Ted Dunning <td...@maprtech.com>
> >> wrote:
> >>
> >>> I am very sympathetic here. Also, somewhat linguistically challenged
> >>>on
> >>> this point since there is a fine line to be walked. All suggestions
> >>>are
> >>> welcome.
> >>>
> >>> How should I answer this? The question was "how can I get alerts for
> >> my
> >>> hbase cluster"?
> >>>
> >>> One answer is definitely MapR. Is there a way to say that without
> >>>being a
> >>> excessively pluggy?
> >>>
> >>> Another answer that I want to underscore is "MapR supports Hbase. A
> >> lot."
> >>>
> >>
> >> The issue is that I might be then tempted to start arguing that Cloudera
> >> Enterprise supports HBase better than MapR. Then we devolve into an
> >>annoying
> >> vendor war which doesn't help anyone.
> >>
> >> Best to just set a policy and stick to it for public responses. I don't
> >>see
> >> anything wrong with your replying off-list to the requester with a sales
> >> pitch. This provides the information that might help the user without
> >> risking polluting the -user list with a lot of discussion of commercial
> >> products.
> >>
> >> -Todd
> >>
> >>
> >>>
> >>> On Mon, Jul 25, 2011 at 11:09 AM, Stack <st...@duboce.net> wrote:
> >>>
> >>> > On Mon, Jul 25, 2011 at 8:54 AM, Ted Dunning
> >> <td...@maprtech.com>
> >>> > wrote:
> >>> > > Slightly off topic, but MapR runs Hbase very handily (several
> >> times
> >>> > faster,
> >>> > > in fact) and provides comprehensive monitoring and alerting out
> >> of the
> >>> > box.
> >>> > >
> >>> >
> >>> > Hey Ted:
> >>> >
> >>> > MapR is good stuff indeed but the above can be read as a raw plug
> >>>for
> >>> > a non-open-source/commercial product.
> >>> >
> >>> > I'd like to petition that you go easy with messages that could
> >>> > possibly be interpreted so. Other, not-such-close-in-friends of
> >>> > hbase, seeing 'commerical' messages up on our list might take it as
> >>> > license to dump their commercial messages for tech related, or not,
> >>> > into hbase mailing lists. A list riddled with commerical messages
> >>> > would likely sour many who are subscribed here.
> >>> >
> >>> > Thanks boss,
> >>> > St.Ack
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Monitoring
Posted by Doug Meil <do...@explorysmedical.com>.
Andrew: nicely put.
Jeff W: I agree with your ordering.
Stack: I agree with the book change for inclusion, and I agree with the
caveat about 'free as in free beer'.
Todd/Ryan: regarding the book reference to CDH, I think the CDH reference
being 'free as in beer' reference is an important differentiator, if it
wasn't it would be a different story.
Ted: re: "My only test for these would be something that helps hbase
users", I agree with the intent, as long as the available license isn't
only commercial.
On 7/25/11 8:43 PM, "Andrew Purtell" <ap...@apache.org> wrote:
>I agree it's a fine line. As far as vendor specific pronouncements, may I
>suggest a little goes a long way, quality over quantity.
>
>In my opinion it's never sufficient to say only "nonopen product Foo does
>X" on an open source project's user list. Instead you need to first
>explain how to do X with available FOSS tools. After that, mentioning
>that Foo does it out of the box and is therefore easier than the
>explained FOSS option is totally acceptable.
>
>Best regards,
>
>
> - Andy
>
>Problems worthy of attack prove their worth by hitting back. - Piet Hein
>(via Tom White)
>
>
>----- Original Message -----
>> From: Todd Lipcon <to...@cloudera.com>
>> To: user@hbase.apache.org
>> Cc:
>> Sent: Monday, July 25, 2011 11:45 AM
>> Subject: Re: Monitoring
>>
>> On Mon, Jul 25, 2011 at 11:28 AM, Ted Dunning <td...@maprtech.com>
>> wrote:
>>
>>> I am very sympathetic here. Also, somewhat linguistically challenged
>>>on
>>> this point since there is a fine line to be walked. All suggestions
>>>are
>>> welcome.
>>>
>>> How should I answer this? The question was "how can I get alerts for
>> my
>>> hbase cluster"?
>>>
>>> One answer is definitely MapR. Is there a way to say that without
>>>being a
>>> excessively pluggy?
>>>
>>> Another answer that I want to underscore is "MapR supports Hbase. A
>> lot."
>>>
>>
>> The issue is that I might be then tempted to start arguing that Cloudera
>> Enterprise supports HBase better than MapR. Then we devolve into an
>>annoying
>> vendor war which doesn't help anyone.
>>
>> Best to just set a policy and stick to it for public responses. I don't
>>see
>> anything wrong with your replying off-list to the requester with a sales
>> pitch. This provides the information that might help the user without
>> risking polluting the -user list with a lot of discussion of commercial
>> products.
>>
>> -Todd
>>
>>
>>>
>>> On Mon, Jul 25, 2011 at 11:09 AM, Stack <st...@duboce.net> wrote:
>>>
>>> > On Mon, Jul 25, 2011 at 8:54 AM, Ted Dunning
>> <td...@maprtech.com>
>>> > wrote:
>>> > > Slightly off topic, but MapR runs Hbase very handily (several
>> times
>>> > faster,
>>> > > in fact) and provides comprehensive monitoring and alerting out
>> of the
>>> > box.
>>> > >
>>> >
>>> > Hey Ted:
>>> >
>>> > MapR is good stuff indeed but the above can be read as a raw plug
>>>for
>>> > a non-open-source/commercial product.
>>> >
>>> > I'd like to petition that you go easy with messages that could
>>> > possibly be interpreted so. Other, not-such-close-in-friends of
>>> > hbase, seeing 'commerical' messages up on our list might take it as
>>> > license to dump their commercial messages for tech related, or not,
>>> > into hbase mailing lists. A list riddled with commerical messages
>>> > would likely sour many who are subscribed here.
>>> >
>>> > Thanks boss,
>>> > St.Ack
>>> >
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
Re: Monitoring
Posted by Andrew Purtell <ap...@apache.org>.
I agree it's a fine line. As far as vendor specific pronouncements, may I suggest a little goes a long way, quality over quantity.
In my opinion it's never sufficient to say only "nonopen product Foo does X" on an open source project's user list. Instead you need to first explain how to do X with available FOSS tools. After that, mentioning that Foo does it out of the box and is therefore easier than the explained FOSS option is totally acceptable.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Todd Lipcon <to...@cloudera.com>
> To: user@hbase.apache.org
> Cc:
> Sent: Monday, July 25, 2011 11:45 AM
> Subject: Re: Monitoring
>
> On Mon, Jul 25, 2011 at 11:28 AM, Ted Dunning <td...@maprtech.com>
> wrote:
>
>> I am very sympathetic here. Also, somewhat linguistically challenged on
>> this point since there is a fine line to be walked. All suggestions are
>> welcome.
>>
>> How should I answer this? The question was "how can I get alerts for
> my
>> hbase cluster"?
>>
>> One answer is definitely MapR. Is there a way to say that without being a
>> excessively pluggy?
>>
>> Another answer that I want to underscore is "MapR supports Hbase. A
> lot."
>>
>
> The issue is that I might be then tempted to start arguing that Cloudera
> Enterprise supports HBase better than MapR. Then we devolve into an annoying
> vendor war which doesn't help anyone.
>
> Best to just set a policy and stick to it for public responses. I don't see
> anything wrong with your replying off-list to the requester with a sales
> pitch. This provides the information that might help the user without
> risking polluting the -user list with a lot of discussion of commercial
> products.
>
> -Todd
>
>
>>
>> On Mon, Jul 25, 2011 at 11:09 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Mon, Jul 25, 2011 at 8:54 AM, Ted Dunning
> <td...@maprtech.com>
>> > wrote:
>> > > Slightly off topic, but MapR runs Hbase very handily (several
> times
>> > faster,
>> > > in fact) and provides comprehensive monitoring and alerting out
> of the
>> > box.
>> > >
>> >
>> > Hey Ted:
>> >
>> > MapR is good stuff indeed but the above can be read as a raw plug for
>> > a non-open-source/commercial product.
>> >
>> > I'd like to petition that you go easy with messages that could
>> > possibly be interpreted so. Other, not-such-close-in-friends of
>> > hbase, seeing 'commerical' messages up on our list might take it as
>> > license to dump their commercial messages for tech related, or not,
>> > into hbase mailing lists. A list riddled with commerical messages
>> > would likely sour many who are subscribed here.
>> >
>> > Thanks boss,
>> > St.Ack
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Monitoring
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jul 25, 2011 at 11:28 AM, Ted Dunning <td...@maprtech.com> wrote:
> I am very sympathetic here. Also, somewhat linguistically challenged on
> this point since there is a fine line to be walked. All suggestions are
> welcome.
>
> How should I answer this? The question was "how can I get alerts for my
> hbase cluster"?
>
> One answer is definitely MapR. Is there a way to say that without being a
> excessively pluggy?
>
> Another answer that I want to underscore is "MapR supports Hbase. A lot."
>
The issue is that I might be then tempted to start arguing that Cloudera
Enterprise supports HBase better than MapR. Then we devolve into an annoying
vendor war which doesn't help anyone.
Best to just set a policy and stick to it for public responses. I don't see
anything wrong with your replying off-list to the requester with a sales
pitch. This provides the information that might help the user without
risking polluting the -user list with a lot of discussion of commercial
products.
-Todd
>
> On Mon, Jul 25, 2011 at 11:09 AM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Jul 25, 2011 at 8:54 AM, Ted Dunning <td...@maprtech.com>
> > wrote:
> > > Slightly off topic, but MapR runs Hbase very handily (several times
> > faster,
> > > in fact) and provides comprehensive monitoring and alerting out of the
> > box.
> > >
> >
> > Hey Ted:
> >
> > MapR is good stuff indeed but the above can be read as a raw plug for
> > a non-open-source/commercial product.
> >
> > I'd like to petition that you go easy with messages that could
> > possibly be interpreted so. Other, not-such-close-in-friends of
> > hbase, seeing 'commerical' messages up on our list might take it as
> > license to dump their commercial messages for tech related, or not,
> > into hbase mailing lists. A list riddled with commerical messages
> > would likely sour many who are subscribed here.
> >
> > Thanks boss,
> > St.Ack
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Monitoring
Posted by Ted Dunning <td...@maprtech.com>.
On Mon, Jul 25, 2011 at 12:00 PM, Stack <st...@duboce.net> wrote:
> I felt you deserved the yellow card because the first response out the
> gate was '(Slightly) off topic' and could be read as a plug for a
> commercial product.
>
Yellow accepted.
> > Another answer that I want to underscore is "MapR supports Hbase. A
> lot."
> >
>
> I know this. And MapR+HBase makes for a sweet story (I'll not go on
> -- someone might pull the yellow card on me). Can we get the word out
> else-wise than as plug-looking messages on this list (I'd like to help
> if I can). Its my sense that such messages rub readers of this list
> the wrong way and might be held against you.
>
It wouldn't be the first time that I rubbed somebody the wrong way. It also
won't be the first time that I have tried to make it good afterwards.
Re: Monitoring
Posted by Stack <st...@duboce.net>.
On Mon, Jul 25, 2011 at 11:28 AM, Ted Dunning <td...@maprtech.com> wrote:
> I am very sympathetic here. Also, somewhat linguistically challenged on
> this point since there is a fine line to be walked. All suggestions are
> welcome.
>
Understood. I'm afraid I'm not known for finesse so its hard to
advise navigating the contours of the fine line. Usually you are good
at it.
> How should I answer this? The question was "how can I get alerts for my
> hbase cluster"?
>
Well, on this list I'd think that folks are asking primarily about
what open source alternatives exist. Once this topic has been
exhausted -- the list is short I think when it comes to this
particular question -- then alternative proprietaries might be
floated.
I felt you deserved the yellow card because the first response out the
gate was '(Slightly) off topic' and could be read as a plug for a
commercial product.
> Another answer that I want to underscore is "MapR supports Hbase. A lot."
>
I know this. And MapR+HBase makes for a sweet story (I'll not go on
-- someone might pull the yellow card on me). Can we get the word out
else-wise than as plug-looking messages on this list (I'd like to help
if I can). Its my sense that such messages rub readers of this list
the wrong way and might be held against you.
Good on you Ted,
St.Ack
Re: Monitoring
Posted by Ted Dunning <td...@maprtech.com>.
I am very sympathetic here. Also, somewhat linguistically challenged on
this point since there is a fine line to be walked. All suggestions are
welcome.
How should I answer this? The question was "how can I get alerts for my
hbase cluster"?
One answer is definitely MapR. Is there a way to say that without being a
excessively pluggy?
Another answer that I want to underscore is "MapR supports Hbase. A lot."
On Mon, Jul 25, 2011 at 11:09 AM, Stack <st...@duboce.net> wrote:
> On Mon, Jul 25, 2011 at 8:54 AM, Ted Dunning <td...@maprtech.com>
> wrote:
> > Slightly off topic, but MapR runs Hbase very handily (several times
> faster,
> > in fact) and provides comprehensive monitoring and alerting out of the
> box.
> >
>
> Hey Ted:
>
> MapR is good stuff indeed but the above can be read as a raw plug for
> a non-open-source/commercial product.
>
> I'd like to petition that you go easy with messages that could
> possibly be interpreted so. Other, not-such-close-in-friends of
> hbase, seeing 'commerical' messages up on our list might take it as
> license to dump their commercial messages for tech related, or not,
> into hbase mailing lists. A list riddled with commerical messages
> would likely sour many who are subscribed here.
>
> Thanks boss,
> St.Ack
>
Re: Monitoring
Posted by Stack <st...@duboce.net>.
On Mon, Jul 25, 2011 at 8:54 AM, Ted Dunning <td...@maprtech.com> wrote:
> Slightly off topic, but MapR runs Hbase very handily (several times faster,
> in fact) and provides comprehensive monitoring and alerting out of the box.
>
Hey Ted:
MapR is good stuff indeed but the above can be read as a raw plug for
a non-open-source/commercial product.
I'd like to petition that you go easy with messages that could
possibly be interpreted so. Other, not-such-close-in-friends of
hbase, seeing 'commerical' messages up on our list might take it as
license to dump their commercial messages for tech related, or not,
into hbase mailing lists. A list riddled with commerical messages
would likely sour many who are subscribed here.
Thanks boss,
St.Ack
Re: Monitoring
Posted by Ted Dunning <td...@maprtech.com>.
Slightly off topic, but MapR runs Hbase very handily (several times faster,
in fact) and provides comprehensive monitoring and alerting out of the box.
Contact me off-list for details if you like.
On Mon, Jul 25, 2011 at 8:09 AM, Joseph Coleman <
joe.coleman@infinitecampus.com> wrote:
> Greetings,
>
> I am relatively new to Hadoop but we now have an 10 node cluster up and
> running just DFS for now and will be expanding this rapidly as well as
> adding Hbase. I am looking to find out what people are using for monitoring
> Hadoop currently. I want to be notified if a node fails, performance
> statistics, failed drive or services ect. I was thinking of using Opsview
> and trying in Ganglia. Thanks in advance
>
> Joe
>
>
Re: Monitoring
Posted by Andrew Purtell <ap...@apache.org>.
Have you tried Ganglia2?
http://sourceforge.net/apps/trac/ganglia/wiki/ganglia-web-2
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Otis Gospodnetic <ot...@yahoo.com>
>To: "user@hbase.apache.org" <us...@hbase.apache.org>
>Sent: Friday, July 29, 2011 5:13 PM
>Subject: Re: Monitoring
>
>Hello,
>
>I was just looking at Ganglia that one of our customers uses for their 200+ node HBase cluster. I do not find Ganglia very nice. For example, I don't think I can make it render some metric for N nodes on a single chart for example, or that I can pick a very specific time period, etc. We at Sematext use our own SPM [1] service to monitor our own HBase and Solr clusters. It doesn't have everything you need (e.g. we only have a prototype for alerts, but we didn't deploy it yet), but the service is currently free for all because we haven't integrated it with any payment services yet..... so whoever is interested.... :)
>
>Otis
>[1] http://sematext.com/spm/index.html
>
>P.S.
>We're looking for people who want to work on SPM - http://sematext.com/about/jobs.html
>
>
>>
>>
>>
>>From: Joseph Coleman <jo...@infinitecampus.com>
>>To: "user@hbase.apache.org" <us...@hbase.apache.org>
>>Sent: Monday, July 25, 2011 11:09 AM
>>Subject: Monitoring
>>
>>Greetings,
>>
>>I am relatively new to Hadoop but we now have an 10 node cluster up and running just DFS for now and will be expanding this rapidly as well as adding Hbase. I am looking to find out what people are using for monitoring Hadoop currently. I want to be notified if a node fails, performance statistics, failed drive or services ect. I was thinking of using Opsview and trying in Ganglia. Thanks in advance
>>
>>Joe
>>
>>
>>
>>
>
>
Re: Monitoring
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
I was just looking at Ganglia that one of our customers uses for their 200+ node HBase cluster. I do not find Ganglia very nice. For example, I don't think I can make it render some metric for N nodes on a single chart for example, or that I can pick a very specific time period, etc. We at Sematext use our own SPM [1] service to monitor our own HBase and Solr clusters. It doesn't have everything you need (e.g. we only have a prototype for alerts, but we didn't deploy it yet), but the service is currently free for all because we haven't integrated it with any payment services yet..... so whoever is interested.... :)
Otis
[1] http://sematext.com/spm/index.html
P.S.
We're looking for people who want to work on SPM - http://sematext.com/about/jobs.html
>
>
>
>From: Joseph Coleman <jo...@infinitecampus.com>
>To: "user@hbase.apache.org" <us...@hbase.apache.org>
>Sent: Monday, July 25, 2011 11:09 AM
>Subject: Monitoring
>
>Greetings,
>
>I am relatively new to Hadoop but we now have an 10 node cluster up and running just DFS for now and will be expanding this rapidly as well as adding Hbase. I am looking to find out what people are using for monitoring Hadoop currently. I want to be notified if a node fails, performance statistics, failed drive or services ect. I was thinking of using Opsview and trying in Ganglia. Thanks in advance
>
>Joe
>
>
>
>