You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2013/11/18 05:01:49 UTC
Re: [DISCUSS] Hama 0.7.0
> I would like to suggest that we solve the messaging scalability issue. WDYT?
The Spilling Queue seems works fine on my cluster. So, if we finish
the below tasks, I think, Hama will be very "Competitive" in terms of
the performance and scalability of the both (pure) BSP and Graph
computing engine.
HAMA-734 Hama Message Manager should be able to delegate the ownership
of internal message queue on request for future superstep.
HAMA-723 Implement sorting in Spilling queue.
HAMA-816 Add the getMsgIterators method for efficient message looping.
HAMA-783 Efficient InMemory Storage for Vertices.
If no objection, i would like to arrange the JIRA tasks for 0.7.0,
based on this. WDYT?
On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com> wrote:
> That looks fine to me. In addition to this, if that task is
> accomplished, are we planning to release a new version (e.g. a minor
> version plus 1)? Just to check as it seems that we have demands on
> frequent releases so that users who need some specific patches can use
> it earlier.
>
>
>
>
> On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
>> According to Suraj's dependency diagram, asynchronous messaging is
>> most important and highest priority for us at the moment. How about we
>> focus on this one? (Of course, some committers can dedicated on doing
>> GPU, ML algorithms, or Interface Refactoring issues, regardless of
>> *core* roadmap).
>>
>> If we agree with this plan, I think we can separate the async
>> messaging into smaller sub-tasks:
>>
>> * Decision of whether we will use existing open source, or not.
>> * Design the asynchronous messaging interface (maybe (spilling)
>> message queue also should be considered together?).
>> * Implementation of asynchronous messaging functions, such as send or flush.
>> * Evaluation and adopt asynchronous messaging as a default messenger system.
>>
>> WDYT?
>>
>>
>> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>>> +1
>>>
>>> BTW, are we going to prioritize tasks in roadmap?
>>>
>>>
>>>
>>>
>>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com> wrote:
>>>> sure, it looks reasonable to me.
>>>> Tommaso
>>>>
>>>>
>>>>
>>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>>
>>>>> Hi all,
>>>>>
>>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
>>>>> version now.
>>>>>
>>>>> I would like to suggest that we solve the messaging scalability issue.
>>>>> WDYT?
>>>>>
>>>>> ...
>>>>>
>>>>> And, according to my experiments, BSP framework shows very nice performance
>>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
>>>>> work on improving the performance of GraphJobRunner.
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: [DISCUSS] Hama 0.7.0
Posted by Chia-Hung Lin <cl...@googlemail.com>.
+1
On 18 November 2013 17:31, Anastasis Andronidis
<an...@hotmail.com> wrote:
> +1 sounds great.
> Anastasis
>
> On 18 Νοε 2013, at 9:01 π.μ., Tommaso Teofili <to...@gmail.com> wrote:
>
>> +1 sounds good to me.
>> Tommaso
>>
>>
>> 2013/11/18 Edward J. Yoon <ed...@apache.org>
>>
>>>> I would like to suggest that we solve the messaging scalability issue.
>>> WDYT?
>>>
>>> The Spilling Queue seems works fine on my cluster. So, if we finish
>>> the below tasks, I think, Hama will be very "Competitive" in terms of
>>> the performance and scalability of the both (pure) BSP and Graph
>>> computing engine.
>>>
>>> HAMA-734 Hama Message Manager should be able to delegate the ownership
>>> of internal message queue on request for future superstep.
>>> HAMA-723 Implement sorting in Spilling queue.
>>> HAMA-816 Add the getMsgIterators method for efficient message looping.
>>> HAMA-783 Efficient InMemory Storage for Vertices.
>>>
>>> If no objection, i would like to arrange the JIRA tasks for 0.7.0,
>>> based on this. WDYT?
>>>
>>> On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com>
>>> wrote:
>>>> That looks fine to me. In addition to this, if that task is
>>>> accomplished, are we planning to release a new version (e.g. a minor
>>>> version plus 1)? Just to check as it seems that we have demands on
>>>> frequent releases so that users who need some specific patches can use
>>>> it earlier.
>>>>
>>>>
>>>>
>>>>
>>>> On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> According to Suraj's dependency diagram, asynchronous messaging is
>>>>> most important and highest priority for us at the moment. How about we
>>>>> focus on this one? (Of course, some committers can dedicated on doing
>>>>> GPU, ML algorithms, or Interface Refactoring issues, regardless of
>>>>> *core* roadmap).
>>>>>
>>>>> If we agree with this plan, I think we can separate the async
>>>>> messaging into smaller sub-tasks:
>>>>>
>>>>> * Decision of whether we will use existing open source, or not.
>>>>> * Design the asynchronous messaging interface (maybe (spilling)
>>>>> message queue also should be considered together?).
>>>>> * Implementation of asynchronous messaging functions, such as send or
>>> flush.
>>>>> * Evaluation and adopt asynchronous messaging as a default messenger
>>> system.
>>>>>
>>>>> WDYT?
>>>>>
>>>>>
>>>>> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com>
>>> wrote:
>>>>>> +1
>>>>>>
>>>>>> BTW, are we going to prioritize tasks in roadmap?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com>
>>> wrote:
>>>>>>> sure, it looks reasonable to me.
>>>>>>> Tommaso
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for
>>> 0.7.0
>>>>>>>> version now.
>>>>>>>>
>>>>>>>> I would like to suggest that we solve the messaging scalability
>>> issue.
>>>>>>>> WDYT?
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> And, according to my experiments, BSP framework shows very nice
>>> performance
>>>>>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll
>>> mainly
>>>>>>>> work on improving the performance of GraphJobRunner.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards, Edward J. Yoon
>>>>>>>> @eddieyoon
>>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>
Re: [DISCUSS] Hama 0.7.0
Posted by Anastasis Andronidis <an...@hotmail.com>.
+1 sounds great.
Anastasis
On 18 Νοε 2013, at 9:01 π.μ., Tommaso Teofili <to...@gmail.com> wrote:
> +1 sounds good to me.
> Tommaso
>
>
> 2013/11/18 Edward J. Yoon <ed...@apache.org>
>
>>> I would like to suggest that we solve the messaging scalability issue.
>> WDYT?
>>
>> The Spilling Queue seems works fine on my cluster. So, if we finish
>> the below tasks, I think, Hama will be very "Competitive" in terms of
>> the performance and scalability of the both (pure) BSP and Graph
>> computing engine.
>>
>> HAMA-734 Hama Message Manager should be able to delegate the ownership
>> of internal message queue on request for future superstep.
>> HAMA-723 Implement sorting in Spilling queue.
>> HAMA-816 Add the getMsgIterators method for efficient message looping.
>> HAMA-783 Efficient InMemory Storage for Vertices.
>>
>> If no objection, i would like to arrange the JIRA tasks for 0.7.0,
>> based on this. WDYT?
>>
>> On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com>
>> wrote:
>>> That looks fine to me. In addition to this, if that task is
>>> accomplished, are we planning to release a new version (e.g. a minor
>>> version plus 1)? Just to check as it seems that we have demands on
>>> frequent releases so that users who need some specific patches can use
>>> it earlier.
>>>
>>>
>>>
>>>
>>> On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
>>>> According to Suraj's dependency diagram, asynchronous messaging is
>>>> most important and highest priority for us at the moment. How about we
>>>> focus on this one? (Of course, some committers can dedicated on doing
>>>> GPU, ML algorithms, or Interface Refactoring issues, regardless of
>>>> *core* roadmap).
>>>>
>>>> If we agree with this plan, I think we can separate the async
>>>> messaging into smaller sub-tasks:
>>>>
>>>> * Decision of whether we will use existing open source, or not.
>>>> * Design the asynchronous messaging interface (maybe (spilling)
>>>> message queue also should be considered together?).
>>>> * Implementation of asynchronous messaging functions, such as send or
>> flush.
>>>> * Evaluation and adopt asynchronous messaging as a default messenger
>> system.
>>>>
>>>> WDYT?
>>>>
>>>>
>>>> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com>
>> wrote:
>>>>> +1
>>>>>
>>>>> BTW, are we going to prioritize tasks in roadmap?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com>
>> wrote:
>>>>>> sure, it looks reasonable to me.
>>>>>> Tommaso
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for
>> 0.7.0
>>>>>>> version now.
>>>>>>>
>>>>>>> I would like to suggest that we solve the messaging scalability
>> issue.
>>>>>>> WDYT?
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> And, according to my experiments, BSP framework shows very nice
>> performance
>>>>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll
>> mainly
>>>>>>> work on improving the performance of GraphJobRunner.
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards, Edward J. Yoon
>>>>>>> @eddieyoon
>>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
Re: [DISCUSS] Hama 0.7.0
Posted by Tommaso Teofili <to...@gmail.com>.
+1 sounds good to me.
Tommaso
2013/11/18 Edward J. Yoon <ed...@apache.org>
> > I would like to suggest that we solve the messaging scalability issue.
> WDYT?
>
> The Spilling Queue seems works fine on my cluster. So, if we finish
> the below tasks, I think, Hama will be very "Competitive" in terms of
> the performance and scalability of the both (pure) BSP and Graph
> computing engine.
>
> HAMA-734 Hama Message Manager should be able to delegate the ownership
> of internal message queue on request for future superstep.
> HAMA-723 Implement sorting in Spilling queue.
> HAMA-816 Add the getMsgIterators method for efficient message looping.
> HAMA-783 Efficient InMemory Storage for Vertices.
>
> If no objection, i would like to arrange the JIRA tasks for 0.7.0,
> based on this. WDYT?
>
> On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> > That looks fine to me. In addition to this, if that task is
> > accomplished, are we planning to release a new version (e.g. a minor
> > version plus 1)? Just to check as it seems that we have demands on
> > frequent releases so that users who need some specific patches can use
> > it earlier.
> >
> >
> >
> >
> > On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
> >> According to Suraj's dependency diagram, asynchronous messaging is
> >> most important and highest priority for us at the moment. How about we
> >> focus on this one? (Of course, some committers can dedicated on doing
> >> GPU, ML algorithms, or Interface Refactoring issues, regardless of
> >> *core* roadmap).
> >>
> >> If we agree with this plan, I think we can separate the async
> >> messaging into smaller sub-tasks:
> >>
> >> * Decision of whether we will use existing open source, or not.
> >> * Design the asynchronous messaging interface (maybe (spilling)
> >> message queue also should be considered together?).
> >> * Implementation of asynchronous messaging functions, such as send or
> flush.
> >> * Evaluation and adopt asynchronous messaging as a default messenger
> system.
> >>
> >> WDYT?
> >>
> >>
> >> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> >>> +1
> >>>
> >>> BTW, are we going to prioritize tasks in roadmap?
> >>>
> >>>
> >>>
> >>>
> >>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com>
> wrote:
> >>>> sure, it looks reasonable to me.
> >>>> Tommaso
> >>>>
> >>>>
> >>>>
> >>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for
> 0.7.0
> >>>>> version now.
> >>>>>
> >>>>> I would like to suggest that we solve the messaging scalability
> issue.
> >>>>> WDYT?
> >>>>>
> >>>>> ...
> >>>>>
> >>>>> And, according to my experiments, BSP framework shows very nice
> performance
> >>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll
> mainly
> >>>>> work on improving the performance of GraphJobRunner.
> >>>>>
> >>>>> --
> >>>>> Best Regards, Edward J. Yoon
> >>>>> @eddieyoon
> >>>>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>