You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by arvind subramanian <al...@gmail.com> on 2010/04/27 22:40:24 UTC
Exploring chukwa-initial questions!
Hello,
Firstly, Congrats on the release of 0.4!
I am Arvind, and I have just got started in exploring Chukwa
I have been reading the docs from the release in 0.40 , issues on JIRA, and
some of the activity in the mailing list.
I am familiar with Hadoop/Hdfs having used them in pseudo-distributed mode
on laptop(Ubuntu 9.10), and have also used hadoop AMI on Ec2.
Has a Chukwa cluster been deployed on Ec2 ? If so, it would be great to see
some documentation/tutorials/pointers of the same, so that I could try it
out.
Also,It would be nice to see more elaboration on setting up the development
environment for Chukwa (pseudo-distributed mode)
Regards,
Arvind
Re: Exploring chukwa-initial questions!
Posted by arvind subramanian <al...@gmail.com>.
Thanks a lot for your inputs,everyone!!
That is a fantastic response for my first mail here! :)
I think I would have to spend some more time tinkering around, before
posting more pointed queries
Cheers,
Arvind
On Tue, Apr 27, 2010 at 8:40 PM, Jiaqi Tan <ta...@gmail.com> wrote:
> Hi Arvind,
>
> Welcome to the mailing list :)
>
> I have been using Chukwa slightly differently from most; I have been
> using it in an "offline" mode to process Hadoop logs and other system
> metrics by using the backfill loader, which is a special mode for
> loading logs previously generated.
>
> I work on failure diagnosis and visualization of system behavior for
> Hadoop, and I have been using Chukwa to help with that. Some of our
> previous work has already been implemented and is available in Chukwa,
> and I have been working on putting more of the stuff in (albeit at a
> much slower rate than I would like).
>
> Jiaqi
>
> On Wed, Apr 28, 2010 at 5:28 AM, Ariel Rabkin <as...@gmail.com> wrote:
> > My lab-mates use chukwa regularly on EC2 to monitor test deployments.
> > Nothing particularly distinctive about it.
> >
> > Our deployment model is typically to run the agents in the cloud, and
> > the collector at Berkeley, pointing to a static HDFS cluster holding
> > the collected logs.
> >
> > --Ari
> >
> > On Tue, Apr 27, 2010 at 1:40 PM, arvind subramanian
> > <al...@gmail.com> wrote:
> >> Hello,
> >>
> >> Firstly, Congrats on the release of 0.4!
> >>
> >> I am Arvind, and I have just got started in exploring Chukwa
> >>
> >> I have been reading the docs from the release in 0.40 , issues on JIRA,
> and
> >> some of the activity in the mailing list.
> >>
> >> I am familiar with Hadoop/Hdfs having used them in pseudo-distributed
> mode
> >> on laptop(Ubuntu 9.10), and have also used hadoop AMI on Ec2.
> >>
> >> Has a Chukwa cluster been deployed on Ec2 ? If so, it would be great to
> see
> >> some documentation/tutorials/pointers of the same, so that I could try
> it
> >> out.
> >>
> >> Also,It would be nice to see more elaboration on setting up the
> development
> >> environment for Chukwa (pseudo-distributed mode)
> >>
> >> Regards,
> >> Arvind
> >>
> >
> >
> >
> > --
> > Ari Rabkin asrabkin@gmail.com
> > UC Berkeley Computer Science Department
> >
>
--
Arvind Subramanian
Re: Exploring chukwa-initial questions!
Posted by Jiaqi Tan <ta...@gmail.com>.
Hi Arvind,
Welcome to the mailing list :)
I have been using Chukwa slightly differently from most; I have been
using it in an "offline" mode to process Hadoop logs and other system
metrics by using the backfill loader, which is a special mode for
loading logs previously generated.
I work on failure diagnosis and visualization of system behavior for
Hadoop, and I have been using Chukwa to help with that. Some of our
previous work has already been implemented and is available in Chukwa,
and I have been working on putting more of the stuff in (albeit at a
much slower rate than I would like).
Jiaqi
On Wed, Apr 28, 2010 at 5:28 AM, Ariel Rabkin <as...@gmail.com> wrote:
> My lab-mates use chukwa regularly on EC2 to monitor test deployments.
> Nothing particularly distinctive about it.
>
> Our deployment model is typically to run the agents in the cloud, and
> the collector at Berkeley, pointing to a static HDFS cluster holding
> the collected logs.
>
> --Ari
>
> On Tue, Apr 27, 2010 at 1:40 PM, arvind subramanian
> <al...@gmail.com> wrote:
>> Hello,
>>
>> Firstly, Congrats on the release of 0.4!
>>
>> I am Arvind, and I have just got started in exploring Chukwa
>>
>> I have been reading the docs from the release in 0.40 , issues on JIRA, and
>> some of the activity in the mailing list.
>>
>> I am familiar with Hadoop/Hdfs having used them in pseudo-distributed mode
>> on laptop(Ubuntu 9.10), and have also used hadoop AMI on Ec2.
>>
>> Has a Chukwa cluster been deployed on Ec2 ? If so, it would be great to see
>> some documentation/tutorials/pointers of the same, so that I could try it
>> out.
>>
>> Also,It would be nice to see more elaboration on setting up the development
>> environment for Chukwa (pseudo-distributed mode)
>>
>> Regards,
>> Arvind
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>
Re: Exploring chukwa-initial questions!
Posted by Ariel Rabkin <as...@gmail.com>.
My lab-mates use chukwa regularly on EC2 to monitor test deployments.
Nothing particularly distinctive about it.
Our deployment model is typically to run the agents in the cloud, and
the collector at Berkeley, pointing to a static HDFS cluster holding
the collected logs.
--Ari
On Tue, Apr 27, 2010 at 1:40 PM, arvind subramanian
<al...@gmail.com> wrote:
> Hello,
>
> Firstly, Congrats on the release of 0.4!
>
> I am Arvind, and I have just got started in exploring Chukwa
>
> I have been reading the docs from the release in 0.40 , issues on JIRA, and
> some of the activity in the mailing list.
>
> I am familiar with Hadoop/Hdfs having used them in pseudo-distributed mode
> on laptop(Ubuntu 9.10), and have also used hadoop AMI on Ec2.
>
> Has a Chukwa cluster been deployed on Ec2 ? If so, it would be great to see
> some documentation/tutorials/pointers of the same, so that I could try it
> out.
>
> Also,It would be nice to see more elaboration on setting up the development
> environment for Chukwa (pseudo-distributed mode)
>
> Regards,
> Arvind
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: Exploring chukwa-initial questions!
Posted by Jerome Boulon <jb...@netflix.com>.
Hi Arvind, Welcome to Chukwa ML ;-)
I'm running Chukwa in a production environment (Netflix) on EC2.
I'm using 4 small instances to collect around 130M events/day
Deploying on EC2 is the same as deploying in in a standard data center.
Except for the fact that you can loose your instance ... For that I'm
running a non-public discovery service.
Do you have any specific questions?
Regards,
/Jerome.
On 4/27/10 1:40 PM, "arvind subramanian" <al...@gmail.com> wrote:
> Hello,
>
> Firstly, Congrats on the release of 0.4!
>
> I am Arvind, and I have just got started in exploring Chukwa
>
> I have been reading the docs from the release in 0.40 , issues on JIRA, and
> some of the activity in the mailing list.
>
> I am familiar with Hadoop/Hdfs having used them in pseudo-distributed mode
> on laptop(Ubuntu 9.10), and have also used hadoop AMI on Ec2.
>
> Has a Chukwa cluster been deployed on Ec2 ? If so, it would be great to see
> some documentation/tutorials/pointers of the same, so that I could try it
> out.
>
> Also,It would be nice to see more elaboration on setting up the development
> environment for Chukwa (pseudo-distributed mode)
>
> Regards,
> Arvind