You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by Eric Yang <er...@gmail.com> on 2010/10/10 03:47:41 UTC
Experiment with Chukwa trunk
Hi,
I have written some instructions on how to deploy Chukwa 0.5
pre-release system with HBase. Instructions are available here:
http://wiki.apache.org/hadoop/Chukwa_Quick_Start
Suggestions and feedback are welcome.
regards,
Eric
Re: Experiment with Chukwa trunk
Posted by Bill Graham <bi...@gmail.com>.
> Hence, I am making Hbase a required component for trunk for a full end to end deployment.
Does that mean it's not required for a minimal end to end deployment? :)
I think it's fine to have HICC require HBase, and I hope to use it at
some point. But I think users should still be able to use the
agent/collector/demux part of Chukwa without requiring HBase if they
don't need that functionality.
On Mon, Oct 11, 2010 at 4:49 PM, Ariel Rabkin <as...@gmail.com> wrote:
> I think I'm with Bill on this.
>
> There are useful Chukwa deployments without HBase, and I don't want to
> scare anybody away by saying "a full deployment has HBase."
>
> I would say "you can use Chukwa on either HDFS or HBase. HBase has
> lower latency, and requires Hadoop 0.20. Using HDFS+Mapred works with
> 0.18"
>
> --Ari
>
> On Mon, Oct 11, 2010 at 4:31 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> For streaming data to Hbase 0.20.6, Hadoop 0.20 is required. If you are
>> using the sequence file and mapreduce, then Hadoop 0.18+ should work fine.
>> I would like to move forward with map reduce on table instead of map reduce
>> on file to reduce data processing latency. Hence, I am making Hbase a
>> required component for trunk for a full end to end deployment. The setup
>> may sound difficult to do, but it is trivial to setup. In the long run,
>> demux does not need to be a map reduce job, and demux will only get
>> optimized for incremental updates, when developers start to adopt the table
>> approach. Considering this migration as planning for a better future. :)
>> For now, the existing Chukwa 0.4 approach is still works in trunk but I am
>> not enhancing the batch model.
>>
>> Regards,
>> Eric
>>
>> On 10/11/10 1:11 PM, "Bill Graham" <bi...@gmail.com> wrote:
>>
>>> Hi Eric,
>>>
>>> A read this over and have a few comments:
>>>
>>> - Prerequisites say that Hadoop 0.20+ is required, which I think isn't
>>> entirely true. Agents, collectors and the data processor processes all
>>> can run with 0.18.3. I though HICC was the only thing that required
>>> 0.20, no? Either way, we should clarify which components require what
>>> version.
>>>
>>> - You talk about a minimal Chukwa install including HBase, which also
>>> seems misleading. Will HBase be required to use Chukwa going forward?
>>> If not, no need to make the barrier for entry sound higher than it
>>> needs to be IMO.
>>>
>>> thanks,
>>> Bill
>>>
>>> On Sat, Oct 9, 2010 at 6:47 PM, Eric Yang <er...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I have written some instructions on how to deploy Chukwa 0.5
>>>> pre-release system with HBase. Instructions are available here:
>>>>
>>>> http://wiki.apache.org/hadoop/Chukwa_Quick_Start
>>>>
>>>> Suggestions and feedback are welcome.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>
>>
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>
Re: Experiment with Chukwa trunk
Posted by Ariel Rabkin <as...@gmail.com>.
I think I'm with Bill on this.
There are useful Chukwa deployments without HBase, and I don't want to
scare anybody away by saying "a full deployment has HBase."
I would say "you can use Chukwa on either HDFS or HBase. HBase has
lower latency, and requires Hadoop 0.20. Using HDFS+Mapred works with
0.18"
--Ari
On Mon, Oct 11, 2010 at 4:31 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
> For streaming data to Hbase 0.20.6, Hadoop 0.20 is required. If you are
> using the sequence file and mapreduce, then Hadoop 0.18+ should work fine.
> I would like to move forward with map reduce on table instead of map reduce
> on file to reduce data processing latency. Hence, I am making Hbase a
> required component for trunk for a full end to end deployment. The setup
> may sound difficult to do, but it is trivial to setup. In the long run,
> demux does not need to be a map reduce job, and demux will only get
> optimized for incremental updates, when developers start to adopt the table
> approach. Considering this migration as planning for a better future. :)
> For now, the existing Chukwa 0.4 approach is still works in trunk but I am
> not enhancing the batch model.
>
> Regards,
> Eric
>
> On 10/11/10 1:11 PM, "Bill Graham" <bi...@gmail.com> wrote:
>
>> Hi Eric,
>>
>> A read this over and have a few comments:
>>
>> - Prerequisites say that Hadoop 0.20+ is required, which I think isn't
>> entirely true. Agents, collectors and the data processor processes all
>> can run with 0.18.3. I though HICC was the only thing that required
>> 0.20, no? Either way, we should clarify which components require what
>> version.
>>
>> - You talk about a minimal Chukwa install including HBase, which also
>> seems misleading. Will HBase be required to use Chukwa going forward?
>> If not, no need to make the barrier for entry sound higher than it
>> needs to be IMO.
>>
>> thanks,
>> Bill
>>
>> On Sat, Oct 9, 2010 at 6:47 PM, Eric Yang <er...@gmail.com> wrote:
>>> Hi,
>>>
>>> I have written some instructions on how to deploy Chukwa 0.5
>>> pre-release system with HBase. Instructions are available here:
>>>
>>> http://wiki.apache.org/hadoop/Chukwa_Quick_Start
>>>
>>> Suggestions and feedback are welcome.
>>>
>>> regards,
>>> Eric
>>>
>>
>
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: Experiment with Chukwa trunk
Posted by Eric Yang <ey...@yahoo-inc.com>.
For streaming data to Hbase 0.20.6, Hadoop 0.20 is required. If you are
using the sequence file and mapreduce, then Hadoop 0.18+ should work fine.
I would like to move forward with map reduce on table instead of map reduce
on file to reduce data processing latency. Hence, I am making Hbase a
required component for trunk for a full end to end deployment. The setup
may sound difficult to do, but it is trivial to setup. In the long run,
demux does not need to be a map reduce job, and demux will only get
optimized for incremental updates, when developers start to adopt the table
approach. Considering this migration as planning for a better future. :)
For now, the existing Chukwa 0.4 approach is still works in trunk but I am
not enhancing the batch model.
Regards,
Eric
On 10/11/10 1:11 PM, "Bill Graham" <bi...@gmail.com> wrote:
> Hi Eric,
>
> A read this over and have a few comments:
>
> - Prerequisites say that Hadoop 0.20+ is required, which I think isn't
> entirely true. Agents, collectors and the data processor processes all
> can run with 0.18.3. I though HICC was the only thing that required
> 0.20, no? Either way, we should clarify which components require what
> version.
>
> - You talk about a minimal Chukwa install including HBase, which also
> seems misleading. Will HBase be required to use Chukwa going forward?
> If not, no need to make the barrier for entry sound higher than it
> needs to be IMO.
>
> thanks,
> Bill
>
> On Sat, Oct 9, 2010 at 6:47 PM, Eric Yang <er...@gmail.com> wrote:
>> Hi,
>>
>> I have written some instructions on how to deploy Chukwa 0.5
>> pre-release system with HBase. Instructions are available here:
>>
>> http://wiki.apache.org/hadoop/Chukwa_Quick_Start
>>
>> Suggestions and feedback are welcome.
>>
>> regards,
>> Eric
>>
>
Re: Experiment with Chukwa trunk
Posted by Bill Graham <bi...@gmail.com>.
Hi Eric,
A read this over and have a few comments:
- Prerequisites say that Hadoop 0.20+ is required, which I think isn't
entirely true. Agents, collectors and the data processor processes all
can run with 0.18.3. I though HICC was the only thing that required
0.20, no? Either way, we should clarify which components require what
version.
- You talk about a minimal Chukwa install including HBase, which also
seems misleading. Will HBase be required to use Chukwa going forward?
If not, no need to make the barrier for entry sound higher than it
needs to be IMO.
thanks,
Bill
On Sat, Oct 9, 2010 at 6:47 PM, Eric Yang <er...@gmail.com> wrote:
> Hi,
>
> I have written some instructions on how to deploy Chukwa 0.5
> pre-release system with HBase. Instructions are available here:
>
> http://wiki.apache.org/hadoop/Chukwa_Quick_Start
>
> Suggestions and feedback are welcome.
>
> regards,
> Eric
>