You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2011/11/24 14:50:48 UTC
Add counting feature to get the total number of message exchanges
Hi all,
I'd like to add counting feature to get the total number of message
exchanges during one synchronization among BSP tasks. Before leave
barrier in sync() method, each task can report their counts.
This information will be useful to user and, some applications (for
example, the sync cost of current SSSP example can be reduced to 1/3).
Should we add this feature?
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Add counting feature to get the total number of message exchanges
Posted by "Edward J. Yoon" <ed...@apache.org>.
All,
Please check this issue:
https://issues.apache.org/jira/browse/HAMA-486
On Fri, Nov 25, 2011 at 11:16 AM, Chia-Hung Lin <cl...@googlemail.com> wrote:
> +1 for pluggable feature/ hadoop's counter arch
>
> On 25 November 2011 01:05, Thomas Jungblut
> <th...@googlemail.com> wrote:
>> Great, I open an issue for this counter stuff.
>>
>> 2011/11/24 Edward J. Yoon <ed...@apache.org>
>>
>>> >> I think it would be good as a 'pluggable' feature as this counting
>>> could>> impact performances.
>>> and
>>>
>>> > It is difficult to say whether we actually need this.*
>>> > *In case of SSSP it is useful, but as Tommaso says it can degrade
>>>
>>> I agree with you. If possible, let's provide as a optional feature.
>>>
>>> > What do you think of a similar architecture of Counters like in Hadoop?
>>>
>>> Good.
>>>
>>> Currently, our Job client reports only shows the total number of
>>> supersteps. I think that the Job reports needs to be strengthened. :-)
>>>
>>> On Thu, Nov 24, 2011 at 11:05 PM, Thomas Jungblut
>>> <th...@googlemail.com> wrote:
>>> > It is difficult to say whether we actually need this.*
>>> > *In case of SSSP it is useful, but as Tommaso says it can degrade
>>> > performance for every other BSP by increasing the cost of the sync.
>>> >
>>> > What do you think of a similar architecture of Counters like in Hadoop?
>>> > This could contain even more information, e.G. the number of bytes
>>> > transferred etc.
>>> >
>>> > 2011/11/24 Tommaso Teofili <to...@gmail.com>
>>> >
>>> >> Hello Edward,
>>> >>
>>> >> I think it would be good as a 'pluggable' feature as this counting could
>>> >> impact performances.
>>> >> So it'd be useful when explicitly "debugging" while it should be, in my
>>> >> opinion, disabled by default.
>>> >>
>>> >> My 2 cents.
>>> >> Tommaso
>>> >>
>>> >>
>>> >> 2011/11/24 Edward J. Yoon <ed...@apache.org>
>>> >>
>>> >> > Hi all,
>>> >> >
>>> >> > I'd like to add counting feature to get the total number of message
>>> >> > exchanges during one synchronization among BSP tasks. Before leave
>>> >> > barrier in sync() method, each task can report their counts.
>>> >> >
>>> >> > This information will be useful to user and, some applications (for
>>> >> > example, the sync cost of current SSSP example can be reduced to 1/3).
>>> >> > Should we add this feature?
>>> >> >
>>> >> > --
>>> >> > Best Regards, Edward J. Yoon
>>> >> > @eddieyoon
>>> >> >
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Thomas Jungblut
>>> > Berlin <th...@gmail.com>
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>>
>>
>> --
>> Thomas Jungblut
>> Berlin <th...@gmail.com>
>>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Add counting feature to get the total number of message exchanges
Posted by Chia-Hung Lin <cl...@googlemail.com>.
+1 for pluggable feature/ hadoop's counter arch
On 25 November 2011 01:05, Thomas Jungblut
<th...@googlemail.com> wrote:
> Great, I open an issue for this counter stuff.
>
> 2011/11/24 Edward J. Yoon <ed...@apache.org>
>
>> >> I think it would be good as a 'pluggable' feature as this counting
>> could>> impact performances.
>> and
>>
>> > It is difficult to say whether we actually need this.*
>> > *In case of SSSP it is useful, but as Tommaso says it can degrade
>>
>> I agree with you. If possible, let's provide as a optional feature.
>>
>> > What do you think of a similar architecture of Counters like in Hadoop?
>>
>> Good.
>>
>> Currently, our Job client reports only shows the total number of
>> supersteps. I think that the Job reports needs to be strengthened. :-)
>>
>> On Thu, Nov 24, 2011 at 11:05 PM, Thomas Jungblut
>> <th...@googlemail.com> wrote:
>> > It is difficult to say whether we actually need this.*
>> > *In case of SSSP it is useful, but as Tommaso says it can degrade
>> > performance for every other BSP by increasing the cost of the sync.
>> >
>> > What do you think of a similar architecture of Counters like in Hadoop?
>> > This could contain even more information, e.G. the number of bytes
>> > transferred etc.
>> >
>> > 2011/11/24 Tommaso Teofili <to...@gmail.com>
>> >
>> >> Hello Edward,
>> >>
>> >> I think it would be good as a 'pluggable' feature as this counting could
>> >> impact performances.
>> >> So it'd be useful when explicitly "debugging" while it should be, in my
>> >> opinion, disabled by default.
>> >>
>> >> My 2 cents.
>> >> Tommaso
>> >>
>> >>
>> >> 2011/11/24 Edward J. Yoon <ed...@apache.org>
>> >>
>> >> > Hi all,
>> >> >
>> >> > I'd like to add counting feature to get the total number of message
>> >> > exchanges during one synchronization among BSP tasks. Before leave
>> >> > barrier in sync() method, each task can report their counts.
>> >> >
>> >> > This information will be useful to user and, some applications (for
>> >> > example, the sync cost of current SSSP example can be reduced to 1/3).
>> >> > Should we add this feature?
>> >> >
>> >> > --
>> >> > Best Regards, Edward J. Yoon
>> >> > @eddieyoon
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Thomas Jungblut
>> > Berlin <th...@gmail.com>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Thomas Jungblut
> Berlin <th...@gmail.com>
>
Re: Add counting feature to get the total number of message exchanges
Posted by Thomas Jungblut <th...@googlemail.com>.
Great, I open an issue for this counter stuff.
2011/11/24 Edward J. Yoon <ed...@apache.org>
> >> I think it would be good as a 'pluggable' feature as this counting
> could>> impact performances.
> and
>
> > It is difficult to say whether we actually need this.*
> > *In case of SSSP it is useful, but as Tommaso says it can degrade
>
> I agree with you. If possible, let's provide as a optional feature.
>
> > What do you think of a similar architecture of Counters like in Hadoop?
>
> Good.
>
> Currently, our Job client reports only shows the total number of
> supersteps. I think that the Job reports needs to be strengthened. :-)
>
> On Thu, Nov 24, 2011 at 11:05 PM, Thomas Jungblut
> <th...@googlemail.com> wrote:
> > It is difficult to say whether we actually need this.*
> > *In case of SSSP it is useful, but as Tommaso says it can degrade
> > performance for every other BSP by increasing the cost of the sync.
> >
> > What do you think of a similar architecture of Counters like in Hadoop?
> > This could contain even more information, e.G. the number of bytes
> > transferred etc.
> >
> > 2011/11/24 Tommaso Teofili <to...@gmail.com>
> >
> >> Hello Edward,
> >>
> >> I think it would be good as a 'pluggable' feature as this counting could
> >> impact performances.
> >> So it'd be useful when explicitly "debugging" while it should be, in my
> >> opinion, disabled by default.
> >>
> >> My 2 cents.
> >> Tommaso
> >>
> >>
> >> 2011/11/24 Edward J. Yoon <ed...@apache.org>
> >>
> >> > Hi all,
> >> >
> >> > I'd like to add counting feature to get the total number of message
> >> > exchanges during one synchronization among BSP tasks. Before leave
> >> > barrier in sync() method, each task can report their counts.
> >> >
> >> > This information will be useful to user and, some applications (for
> >> > example, the sync cost of current SSSP example can be reduced to 1/3).
> >> > Should we add this feature?
> >> >
> >> > --
> >> > Best Regards, Edward J. Yoon
> >> > @eddieyoon
> >> >
> >>
> >
> >
> >
> > --
> > Thomas Jungblut
> > Berlin <th...@gmail.com>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Thomas Jungblut
Berlin <th...@gmail.com>
Re: Add counting feature to get the total number of message exchanges
Posted by "Edward J. Yoon" <ed...@apache.org>.
>> I think it would be good as a 'pluggable' feature as this counting could>> impact performances.
and
> It is difficult to say whether we actually need this.*
> *In case of SSSP it is useful, but as Tommaso says it can degrade
I agree with you. If possible, let's provide as a optional feature.
> What do you think of a similar architecture of Counters like in Hadoop?
Good.
Currently, our Job client reports only shows the total number of
supersteps. I think that the Job reports needs to be strengthened. :-)
On Thu, Nov 24, 2011 at 11:05 PM, Thomas Jungblut
<th...@googlemail.com> wrote:
> It is difficult to say whether we actually need this.*
> *In case of SSSP it is useful, but as Tommaso says it can degrade
> performance for every other BSP by increasing the cost of the sync.
>
> What do you think of a similar architecture of Counters like in Hadoop?
> This could contain even more information, e.G. the number of bytes
> transferred etc.
>
> 2011/11/24 Tommaso Teofili <to...@gmail.com>
>
>> Hello Edward,
>>
>> I think it would be good as a 'pluggable' feature as this counting could
>> impact performances.
>> So it'd be useful when explicitly "debugging" while it should be, in my
>> opinion, disabled by default.
>>
>> My 2 cents.
>> Tommaso
>>
>>
>> 2011/11/24 Edward J. Yoon <ed...@apache.org>
>>
>> > Hi all,
>> >
>> > I'd like to add counting feature to get the total number of message
>> > exchanges during one synchronization among BSP tasks. Before leave
>> > barrier in sync() method, each task can report their counts.
>> >
>> > This information will be useful to user and, some applications (for
>> > example, the sync cost of current SSSP example can be reduced to 1/3).
>> > Should we add this feature?
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>>
>
>
>
> --
> Thomas Jungblut
> Berlin <th...@gmail.com>
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Add counting feature to get the total number of message exchanges
Posted by Thomas Jungblut <th...@googlemail.com>.
It is difficult to say whether we actually need this.*
*In case of SSSP it is useful, but as Tommaso says it can degrade
performance for every other BSP by increasing the cost of the sync.
What do you think of a similar architecture of Counters like in Hadoop?
This could contain even more information, e.G. the number of bytes
transferred etc.
2011/11/24 Tommaso Teofili <to...@gmail.com>
> Hello Edward,
>
> I think it would be good as a 'pluggable' feature as this counting could
> impact performances.
> So it'd be useful when explicitly "debugging" while it should be, in my
> opinion, disabled by default.
>
> My 2 cents.
> Tommaso
>
>
> 2011/11/24 Edward J. Yoon <ed...@apache.org>
>
> > Hi all,
> >
> > I'd like to add counting feature to get the total number of message
> > exchanges during one synchronization among BSP tasks. Before leave
> > barrier in sync() method, each task can report their counts.
> >
> > This information will be useful to user and, some applications (for
> > example, the sync cost of current SSSP example can be reduced to 1/3).
> > Should we add this feature?
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
--
Thomas Jungblut
Berlin <th...@gmail.com>
Re: Add counting feature to get the total number of message exchanges
Posted by Tommaso Teofili <to...@gmail.com>.
Hello Edward,
I think it would be good as a 'pluggable' feature as this counting could
impact performances.
So it'd be useful when explicitly "debugging" while it should be, in my
opinion, disabled by default.
My 2 cents.
Tommaso
2011/11/24 Edward J. Yoon <ed...@apache.org>
> Hi all,
>
> I'd like to add counting feature to get the total number of message
> exchanges during one synchronization among BSP tasks. Before leave
> barrier in sync() method, each task can report their counts.
>
> This information will be useful to user and, some applications (for
> example, the sync cost of current SSSP example can be reduced to 1/3).
> Should we add this feature?
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>