You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2013/08/28 05:50:55 UTC

[DISCUSS] Hama 0.7.0

Hi all,

After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
version now.

I would like to suggest that we solve the messaging scalability issue. WDYT?

...

And, according to my experiments, BSP framework shows very nice performance
(I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
work on improving the performance of GraphJobRunner.

-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: [DISCUSS] Hama 0.7.0

Posted by Chia-Hung Lin <cl...@googlemail.com>.
+1

On 18 November 2013 17:31, Anastasis Andronidis
<an...@hotmail.com> wrote:
> +1 sounds great.
> Anastasis
>
> On 18 Νοε 2013, at 9:01 π.μ., Tommaso Teofili <to...@gmail.com> wrote:
>
>> +1 sounds good to me.
>> Tommaso
>>
>>
>> 2013/11/18 Edward J. Yoon <ed...@apache.org>
>>
>>>> I would like to suggest that we solve the messaging scalability issue.
>>> WDYT?
>>>
>>> The Spilling Queue seems works fine on my cluster. So, if we finish
>>> the below tasks, I think, Hama will be very "Competitive" in terms of
>>> the performance and scalability of the both (pure) BSP and Graph
>>> computing engine.
>>>
>>> HAMA-734 Hama Message Manager should be able to delegate the ownership
>>> of internal message queue on request for future superstep.
>>> HAMA-723 Implement sorting in Spilling queue.
>>> HAMA-816 Add the getMsgIterators method for efficient message looping.
>>> HAMA-783 Efficient InMemory Storage for Vertices.
>>>
>>> If no objection, i would like to arrange the JIRA tasks for 0.7.0,
>>> based on this. WDYT?
>>>
>>> On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com>
>>> wrote:
>>>> That looks fine to me. In addition to this, if that task is
>>>> accomplished, are we planning to release a new version (e.g. a minor
>>>> version plus 1)? Just to check as it seems that we have demands on
>>>> frequent releases so that users who need some specific patches can use
>>>> it earlier.
>>>>
>>>>
>>>>
>>>>
>>>> On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
>>>>> According to Suraj's dependency diagram, asynchronous messaging is
>>>>> most important and highest priority for us at the moment. How about we
>>>>> focus on this one? (Of course, some committers can dedicated on doing
>>>>> GPU, ML algorithms, or Interface Refactoring issues, regardless of
>>>>> *core* roadmap).
>>>>>
>>>>> If we agree with this plan, I think we can separate the async
>>>>> messaging into smaller sub-tasks:
>>>>>
>>>>> * Decision of whether we will use existing open source, or not.
>>>>> * Design the asynchronous messaging interface (maybe (spilling)
>>>>> message queue also should be considered together?).
>>>>> * Implementation of asynchronous messaging functions, such as send or
>>> flush.
>>>>> * Evaluation and adopt asynchronous messaging as a default messenger
>>> system.
>>>>>
>>>>> WDYT?
>>>>>
>>>>>
>>>>> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com>
>>> wrote:
>>>>>> +1
>>>>>>
>>>>>> BTW, are we going to prioritize tasks in roadmap?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com>
>>> wrote:
>>>>>>> sure, it looks reasonable to me.
>>>>>>> Tommaso
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for
>>> 0.7.0
>>>>>>>> version now.
>>>>>>>>
>>>>>>>> I would like to suggest that we solve the messaging scalability
>>> issue.
>>>>>>>> WDYT?
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> And, according to my experiments, BSP framework shows very nice
>>> performance
>>>>>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll
>>> mainly
>>>>>>>> work on improving the performance of GraphJobRunner.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards, Edward J. Yoon
>>>>>>>> @eddieyoon
>>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>

Re: [DISCUSS] Hama 0.7.0

Posted by Anastasis Andronidis <an...@hotmail.com>.
+1 sounds great.
Anastasis

On 18 Νοε 2013, at 9:01 π.μ., Tommaso Teofili <to...@gmail.com> wrote:

> +1 sounds good to me.
> Tommaso
> 
> 
> 2013/11/18 Edward J. Yoon <ed...@apache.org>
> 
>>> I would like to suggest that we solve the messaging scalability issue.
>> WDYT?
>> 
>> The Spilling Queue seems works fine on my cluster. So, if we finish
>> the below tasks, I think, Hama will be very "Competitive" in terms of
>> the performance and scalability of the both (pure) BSP and Graph
>> computing engine.
>> 
>> HAMA-734 Hama Message Manager should be able to delegate the ownership
>> of internal message queue on request for future superstep.
>> HAMA-723 Implement sorting in Spilling queue.
>> HAMA-816 Add the getMsgIterators method for efficient message looping.
>> HAMA-783 Efficient InMemory Storage for Vertices.
>> 
>> If no objection, i would like to arrange the JIRA tasks for 0.7.0,
>> based on this. WDYT?
>> 
>> On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com>
>> wrote:
>>> That looks fine to me. In addition to this, if that task is
>>> accomplished, are we planning to release a new version (e.g. a minor
>>> version plus 1)? Just to check as it seems that we have demands on
>>> frequent releases so that users who need some specific patches can use
>>> it earlier.
>>> 
>>> 
>>> 
>>> 
>>> On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
>>>> According to Suraj's dependency diagram, asynchronous messaging is
>>>> most important and highest priority for us at the moment. How about we
>>>> focus on this one? (Of course, some committers can dedicated on doing
>>>> GPU, ML algorithms, or Interface Refactoring issues, regardless of
>>>> *core* roadmap).
>>>> 
>>>> If we agree with this plan, I think we can separate the async
>>>> messaging into smaller sub-tasks:
>>>> 
>>>> * Decision of whether we will use existing open source, or not.
>>>> * Design the asynchronous messaging interface (maybe (spilling)
>>>> message queue also should be considered together?).
>>>> * Implementation of asynchronous messaging functions, such as send or
>> flush.
>>>> * Evaluation and adopt asynchronous messaging as a default messenger
>> system.
>>>> 
>>>> WDYT?
>>>> 
>>>> 
>>>> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com>
>> wrote:
>>>>> +1
>>>>> 
>>>>> BTW, are we going to prioritize tasks in roadmap?
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com>
>> wrote:
>>>>>> sure, it looks reasonable to me.
>>>>>> Tommaso
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for
>> 0.7.0
>>>>>>> version now.
>>>>>>> 
>>>>>>> I would like to suggest that we solve the messaging scalability
>> issue.
>>>>>>> WDYT?
>>>>>>> 
>>>>>>> ...
>>>>>>> 
>>>>>>> And, according to my experiments, BSP framework shows very nice
>> performance
>>>>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll
>> mainly
>>>>>>> work on improving the performance of GraphJobRunner.
>>>>>>> 
>>>>>>> --
>>>>>>> Best Regards, Edward J. Yoon
>>>>>>> @eddieyoon
>>>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>> 
>> 
>> 
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>> 


Re: [DISCUSS] Hama 0.7.0

Posted by Tommaso Teofili <to...@gmail.com>.
+1 sounds good to me.
Tommaso


2013/11/18 Edward J. Yoon <ed...@apache.org>

> > I would like to suggest that we solve the messaging scalability issue.
> WDYT?
>
> The Spilling Queue seems works fine on my cluster. So, if we finish
> the below tasks, I think, Hama will be very "Competitive" in terms of
> the performance and scalability of the both (pure) BSP and Graph
> computing engine.
>
> HAMA-734 Hama Message Manager should be able to delegate the ownership
> of internal message queue on request for future superstep.
> HAMA-723 Implement sorting in Spilling queue.
> HAMA-816 Add the getMsgIterators method for efficient message looping.
> HAMA-783 Efficient InMemory Storage for Vertices.
>
> If no objection, i would like to arrange the JIRA tasks for 0.7.0,
> based on this. WDYT?
>
> On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> > That looks fine to me. In addition to this, if that task is
> > accomplished, are we planning to release a new version (e.g. a minor
> > version plus 1)? Just to check as it seems that we have demands on
> > frequent releases so that users who need some specific patches can use
> > it earlier.
> >
> >
> >
> >
> > On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
> >> According to Suraj's dependency diagram, asynchronous messaging is
> >> most important and highest priority for us at the moment. How about we
> >> focus on this one? (Of course, some committers can dedicated on doing
> >> GPU, ML algorithms, or Interface Refactoring issues, regardless of
> >> *core* roadmap).
> >>
> >> If we agree with this plan, I think we can separate the async
> >> messaging into smaller sub-tasks:
> >>
> >> * Decision of whether we will use existing open source, or not.
> >> * Design the asynchronous messaging interface (maybe (spilling)
> >> message queue also should be considered together?).
> >> * Implementation of asynchronous messaging functions, such as send or
> flush.
> >> * Evaluation and adopt asynchronous messaging as a default messenger
> system.
> >>
> >> WDYT?
> >>
> >>
> >> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> >>> +1
> >>>
> >>> BTW, are we going to prioritize tasks in roadmap?
> >>>
> >>>
> >>>
> >>>
> >>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com>
> wrote:
> >>>> sure, it looks reasonable to me.
> >>>> Tommaso
> >>>>
> >>>>
> >>>>
> >>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for
> 0.7.0
> >>>>> version now.
> >>>>>
> >>>>> I would like to suggest that we solve the messaging scalability
> issue.
> >>>>> WDYT?
> >>>>>
> >>>>> ...
> >>>>>
> >>>>> And, according to my experiments, BSP framework shows very nice
> performance
> >>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll
> mainly
> >>>>> work on improving the performance of GraphJobRunner.
> >>>>>
> >>>>> --
> >>>>> Best Regards, Edward J. Yoon
> >>>>> @eddieyoon
> >>>>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Re: [DISCUSS] Hama 0.7.0

Posted by "Edward J. Yoon" <ed...@apache.org>.
> I would like to suggest that we solve the messaging scalability issue. WDYT?

The Spilling Queue seems works fine on my cluster. So, if we finish
the below tasks, I think, Hama will be very "Competitive" in terms of
the performance and scalability of the both (pure) BSP and Graph
computing engine.

HAMA-734 Hama Message Manager should be able to delegate the ownership
of internal message queue on request for future superstep.
HAMA-723 Implement sorting in Spilling queue.
HAMA-816 Add the getMsgIterators method for efficient message looping.
HAMA-783 Efficient InMemory Storage for Vertices.

If no objection, i would like to arrange the JIRA tasks for 0.7.0,
based on this. WDYT?

On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin <cl...@googlemail.com> wrote:
> That looks fine to me. In addition to this, if that task is
> accomplished, are we planning to release a new version (e.g. a minor
> version plus 1)? Just to check as it seems that we have demands on
> frequent releases so that users who need some specific patches can use
> it earlier.
>
>
>
>
> On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
>> According to Suraj's dependency diagram, asynchronous messaging is
>> most important and highest priority for us at the moment. How about we
>> focus on this one? (Of course, some committers can dedicated on doing
>> GPU, ML algorithms, or Interface Refactoring issues, regardless of
>> *core* roadmap).
>>
>> If we agree with this plan, I think we can separate the async
>> messaging into smaller sub-tasks:
>>
>> * Decision of whether we will use existing open source, or not.
>> * Design the asynchronous messaging interface (maybe (spilling)
>> message queue also should be considered together?).
>> * Implementation of asynchronous messaging functions, such as send or flush.
>> * Evaluation and adopt asynchronous messaging as a default messenger system.
>>
>> WDYT?
>>
>>
>> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>>> +1
>>>
>>> BTW, are we going to prioritize tasks in roadmap?
>>>
>>>
>>>
>>>
>>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com> wrote:
>>>> sure, it looks reasonable to me.
>>>> Tommaso
>>>>
>>>>
>>>>
>>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>>
>>>>> Hi all,
>>>>>
>>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
>>>>> version now.
>>>>>
>>>>> I would like to suggest that we solve the messaging scalability issue.
>>>>> WDYT?
>>>>>
>>>>> ...
>>>>>
>>>>> And, according to my experiments, BSP framework shows very nice performance
>>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
>>>>> work on improving the performance of GraphJobRunner.
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: [DISCUSS] Hama 0.7.0

Posted by Chia-Hung Lin <cl...@googlemail.com>.
That looks fine to me. In addition to this, if that task is
accomplished, are we planning to release a new version (e.g. a minor
version plus 1)? Just to check as it seems that we have demands on
frequent releases so that users who need some specific patches can use
it earlier.




On 3 September 2013 09:45, Edward J. Yoon <ed...@apache.org> wrote:
> According to Suraj's dependency diagram, asynchronous messaging is
> most important and highest priority for us at the moment. How about we
> focus on this one? (Of course, some committers can dedicated on doing
> GPU, ML algorithms, or Interface Refactoring issues, regardless of
> *core* roadmap).
>
> If we agree with this plan, I think we can separate the async
> messaging into smaller sub-tasks:
>
> * Decision of whether we will use existing open source, or not.
> * Design the asynchronous messaging interface (maybe (spilling)
> message queue also should be considered together?).
> * Implementation of asynchronous messaging functions, such as send or flush.
> * Evaluation and adopt asynchronous messaging as a default messenger system.
>
> WDYT?
>
>
> On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>> +1
>>
>> BTW, are we going to prioritize tasks in roadmap?
>>
>>
>>
>>
>> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com> wrote:
>>> sure, it looks reasonable to me.
>>> Tommaso
>>>
>>>
>>>
>>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>>
>>>> Hi all,
>>>>
>>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
>>>> version now.
>>>>
>>>> I would like to suggest that we solve the messaging scalability issue.
>>>> WDYT?
>>>>
>>>> ...
>>>>
>>>> And, according to my experiments, BSP framework shows very nice performance
>>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
>>>> work on improving the performance of GraphJobRunner.
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon

Re: [DISCUSS] Hama 0.7.0

Posted by "Edward J. Yoon" <ed...@apache.org>.
According to Suraj's dependency diagram, asynchronous messaging is
most important and highest priority for us at the moment. How about we
focus on this one? (Of course, some committers can dedicated on doing
GPU, ML algorithms, or Interface Refactoring issues, regardless of
*core* roadmap).

If we agree with this plan, I think we can separate the async
messaging into smaller sub-tasks:

* Decision of whether we will use existing open source, or not.
* Design the asynchronous messaging interface (maybe (spilling)
message queue also should be considered together?).
* Implementation of asynchronous messaging functions, such as send or flush.
* Evaluation and adopt asynchronous messaging as a default messenger system.

WDYT?


On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin <cl...@googlemail.com> wrote:
> +1
>
> BTW, are we going to prioritize tasks in roadmap?
>
>
>
>
> On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com> wrote:
>> sure, it looks reasonable to me.
>> Tommaso
>>
>>
>>
>> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>>
>>> Hi all,
>>>
>>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
>>> version now.
>>>
>>> I would like to suggest that we solve the messaging scalability issue.
>>> WDYT?
>>>
>>> ...
>>>
>>> And, according to my experiments, BSP framework shows very nice performance
>>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
>>> work on improving the performance of GraphJobRunner.
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: [DISCUSS] Hama 0.7.0

Posted by Chia-Hung Lin <cl...@googlemail.com>.
+1

BTW, are we going to prioritize tasks in roadmap?




On 28 August 2013 14:17, Tommaso Teofili <to...@gmail.com> wrote:
> sure, it looks reasonable to me.
> Tommaso
>
>
>
> 2013/8/28 Edward J. Yoon <ed...@apache.org>
>
>> Hi all,
>>
>> After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
>> version now.
>>
>> I would like to suggest that we solve the messaging scalability issue.
>> WDYT?
>>
>> ...
>>
>> And, according to my experiments, BSP framework shows very nice performance
>> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
>> work on improving the performance of GraphJobRunner.
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>

Re: [DISCUSS] Hama 0.7.0

Posted by Tommaso Teofili <to...@gmail.com>.
sure, it looks reasonable to me.
Tommaso



2013/8/28 Edward J. Yoon <ed...@apache.org>

> Hi all,
>
> After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0
> version now.
>
> I would like to suggest that we solve the messaging scalability issue.
> WDYT?
>
> ...
>
> And, according to my experiments, BSP framework shows very nice performance
> (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly
> work on improving the performance of GraphJobRunner.
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>