Counters

[Bernd Fondermann <be...@gmail.com>]

Hi,

getting more familiar with Twill, questions keep coming up.
For example: One of the great things about MapRed is Counters.
Is there a similar concept (yet) at Twill? Or do you recommend using Curator?

Thanks,

  Bernd

Re: Counters

[Terence Yim <ch...@gmail.com>]

Hi Bernd,

By "metrics report" I mean a way for TwillRunnable to emit metrics
(e.g. increment a named counter by n, which basically is Counter) and
have way to aggregate across all runnable instances of a given Twill
application.
Currently there is no such thing in Twill, but I think it is valuable
to have it.

Terence

On Wed, Jan 22, 2014 at 7:40 AM, Bernd Fondermann
<be...@gmail.com> wrote:
> On Tue, Jan 21, 2014 at 10:02 PM, Terence Yim <ch...@gmail.com> wrote:
>> Hi Bernd,
>>
>> Is the counter concept that you talk about something like metrics
>> report? We've thought about it, but not yet implemented.
>
> I'm talking about the Distributed Counter concept in classic MapReduce
> where all MR tasks for the same job can for example count overall bad
> records and such. It's like what Curator's Distributed Atomic Long[1]
> provides, without the subtract.
>
> I could not find a reference to "metrics report" in Twill, but I'm
> familiar with Hadoop's metrics.
>
>    Bernd
>
> [1] http://curator.apache.org/curator-recipes/distributed-atomic-long.html
>
>> Potentially
>> it could be done by leveraging the same Kafka instance for logging to
>> transport custom metrics from containers as well. I think it's
>> something valuable to have in Twill.
>>
>> Thanks,
>> Terence
>>
>> On Tue, Jan 21, 2014 at 1:23 AM, Bernd Fondermann
>> <be...@gmail.com> wrote:
>>> Hi,
>>>
>>> getting more familiar with Twill, questions keep coming up.
>>> For example: One of the great things about MapRed is Counters.
>>> Is there a similar concept (yet) at Twill? Or do you recommend using Curator?
>>>
>>> Thanks,
>>>
>>>   Bernd

Re: Counters

[Bernd Fondermann <be...@gmail.com>]

On Tue, Jan 21, 2014 at 10:02 PM, Terence Yim <ch...@gmail.com> wrote:
> Hi Bernd,
>
> Is the counter concept that you talk about something like metrics
> report? We've thought about it, but not yet implemented.

I'm talking about the Distributed Counter concept in classic MapReduce
where all MR tasks for the same job can for example count overall bad
records and such. It's like what Curator's Distributed Atomic Long[1]
provides, without the subtract.

I could not find a reference to "metrics report" in Twill, but I'm
familiar with Hadoop's metrics.

   Bernd

[1] http://curator.apache.org/curator-recipes/distributed-atomic-long.html

> Potentially
> it could be done by leveraging the same Kafka instance for logging to
> transport custom metrics from containers as well. I think it's
> something valuable to have in Twill.
>
> Thanks,
> Terence
>
> On Tue, Jan 21, 2014 at 1:23 AM, Bernd Fondermann
> <be...@gmail.com> wrote:
>> Hi,
>>
>> getting more familiar with Twill, questions keep coming up.
>> For example: One of the great things about MapRed is Counters.
>> Is there a similar concept (yet) at Twill? Or do you recommend using Curator?
>>
>> Thanks,
>>
>>   Bernd

Re: Counters

[Terence Yim <ch...@gmail.com>]

Hi Bernd,

Is the counter concept that you talk about something like metrics
report? We've thought about it, but not yet implemented. Potentially
it could be done by leveraging the same Kafka instance for logging to
transport custom metrics from containers as well. I think it's
something valuable to have in Twill.

Thanks,
Terence

On Tue, Jan 21, 2014 at 1:23 AM, Bernd Fondermann
<be...@gmail.com> wrote:
> Hi,
>
> getting more familiar with Twill, questions keep coming up.
> For example: One of the great things about MapRed is Counters.
> Is there a similar concept (yet) at Twill? Or do you recommend using Curator?
>
> Thanks,
>
>   Bernd