You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hama.apache.org by Apache Wiki <wi...@apache.org> on 2010/05/10 08:01:48 UTC
[Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Hama Wiki" for change notification.
The "Architecture" page has been changed by Edward J. Yoon.
http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
--------------------------------------------------
This effort is still a "work in progress". Please feel free to add comments.
----
+
+ == The overall architecture of HAMA ==
+ {{{
+ +-----------------------------------------------+
+ | HAMA APIs |
+ +-----------------------------------------------+
+ --------------------------------------------------------
+ +---------------------------+
+ | Graph Computing Framework |
+ +---------------------------+ Computing Engine
+ +-----------+ +---------------------------------+
+ | MapReduce | | BSP (Bulk Synchronous Parallel) |
+ +-----------+ +---------------------------------+
+ --------------------------------------------------------
+ +-----------------------------------------------+
+ | Zookeeper | Distributed Locking Service
+ +-----------------------------------------------+
+ --------------------------------------------------------
+ +-----------------------------------------------+
+ | |
+ | +-----------------------------+ |
+ | | HBase : Distributed Storage | |
+ | +-----------------------------+ | Distributed Storage Systems
+ | +---------------------------------+ |
+ | | HDFS : Distributed File System | |
+ | +---------------------------------+ |
+ +-----------------------------------------------+
+ }}}
+
== BSP ==
The BSP package is a implementation of BSP over Hadoop RPC(sockets).
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by Hyunsik Choi <hy...@gmail.com>.
I correct typos as follows:
I'm not focusing on which one is better between BSP and M/R. I misunderstood
your intention due to the above your mention. It is because we will need not
only the BSP framework but also other programs to help *users* to develop
matrix algebra/graph processing *applications*.
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Tue, May 11, 2010 at 12:55 PM, Hyunsik Choi <hy...@gmail.com>wrote:
> > As mentioned above, I'd like to show only some example of how you
> implement your own matrix/graph applications using BSP.
>
> I'm not focusing on which one is better between BSP and M/R. I
> misunderstood your intention due to the above your mention. It is because we
> will need not only the BSP framework but also other programs to help your to
> develop matrix algebra/graph processing.
>
> Best regards,
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
> On Tue, May 11, 2010 at 11:19 AM, Edward J. Yoon <ed...@apache.org>wrote:
>
>> In this mailing
>> (
>> http://markmail.org/search/?q=hama-dev+discuss#query:hama-dev%20discuss+page:1+mid:amlvccbptom3yro3+state:results
>> )
>> and others, we discussed about:
>>
>> 1) We can't implement all of matrix algebra. 2) Even if we do that,
>> the code complexity based on M/R model is too high. As we learned, the
>> M/R model isn't fit well for the complex numerical/relational data
>> processing required for many communications between each others. 3)
>> Admittedly, BSP is flexible than M/R.
>>
>> Therefore, We've aligned our goal: to provide easy *matrix/graph
>> computing* programming environment on the distributed system (hadoop),
>> such as M/R framework. (NOT library) So, we won't develop only the BSP
>> framework if there is other solutions.
>>
>> On Tue, May 11, 2010 at 10:29 AM, Hyunsik Choi <hy...@gmail.com>
>> wrote:
>> > According to you, the main objective of hama appears to be developing
>> only
>> > the BSP framework. Is it right?
>> >
>> > --
>> > Hyunsik Choi
>> > Database & Information Systems Group, Korea Univ.
>> > http://diveintodata.org
>> >
>> >
>> > On Mon, May 10, 2010 at 8:34 PM, Edward J. Yoon <edwardyoon@apache.org
>> >wrote:
>> >
>> >> > In the distant future, the graph package would use both MR and BSP.
>> But,
>> >>
>> >> Where will M/R use for?
>> >>
>> >> I'd like to cut M/R out of HAMA project and leave only the pure BSP &
>> >> Graph computing framework as simple as possible. Then, the rest are
>> >> user's matrix/graph applications.
>> >>
>> >> And, I'd like to consider about building an contributor/developer
>> >> ECOsystem.
>> >>
>> >> > now i'll try to develop the graph package with only BSP. How about
>> the
>> >> > matrix package?
>> >>
>> >> As mentioned above, I'd like to show only some example of how you
>> >> implement your own matrix/graph applications using BSP.
>> >>
>> >> I think the block of hadoop families should be added on diagram only
>> >> to help understand relations between hadoop and hama,
>> >>
>> >> On Mon, May 10, 2010 at 7:40 PM, Hyunsik Choi <hy...@gmail.com>
>> >> wrote:
>> >> > In the distant future, the graph package would use both MR and BSP.
>> But,
>> >> > now i'll try to develop the graph package with only BSP. How about
>> the
>> >> > matrix package?
>> >> >
>> >> > --
>> >> > Hyunsik Choi
>> >> > Database & Information Systems Group, Korea Univ.
>> >> > http://diveintodata.org
>> >> >
>> >> >
>> >> > On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <
>> edwardyoon@apache.org
>> >> >wrote:
>> >> >
>> >> >> > zookeeper should be enclosed in BSP since zookeeper is not
>> disclosed
>> >> to
>> >> >> > external. It is only used inside the BSP.
>> >> >>
>> >> >> Right you are. And also, the top of M/R and BSP should be clarified,
>> >> >> but currently something is odd overall.
>> >> >>
>> >> >> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <
>> hyunsik.choi@gmail.com>
>> >> >> wrote:
>> >> >> > Good job! But, the matrix package is omitted. Besides, I think
>> that
>> >> >> > zookeeper should be enclosed in BSP since zookeeper is not
>> disclosed
>> >> to
>> >> >> > external. It is only used inside the BSP.
>> >> >> >
>> >> >> > Best regards,
>> >> >> > --
>> >> >> > Hyunsik Choi
>> >> >> > Database & Information Systems Group, Korea Univ.
>> >> >> > http://diveintodata.org
>> >> >> >
>> >> >> >
>> >> >> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <
>> wikidiffs@apache.org>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Dear Wiki user,
>> >> >> >>
>> >> >> >> You have subscribed to a wiki page or wiki category on "Hama
>> Wiki"
>> >> for
>> >> >> >> change notification.
>> >> >> >>
>> >> >> >> The "Architecture" page has been changed by Edward J. Yoon.
>> >> >> >>
>> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
>> >> >> >>
>> >> >> >> --------------------------------------------------
>> >> >> >>
>> >> >> >>
>> >> >> >> This effort is still a "work in progress". Please feel free to
>> add
>> >> >> >> comments.
>> >> >> >> ----
>> >> >> >> +
>> >> >> >> + == The overall architecture of HAMA ==
>> >> >> >> + {{{
>> >> >> >> + +-----------------------------------------------+
>> >> >> >> + | HAMA APIs |
>> >> >> >> + +-----------------------------------------------+
>> >> >> >> + --------------------------------------------------------
>> >> >> >> + +---------------------------+
>> >> >> >> + | Graph Computing Framework |
>> >> >> >> + +---------------------------+
>> Computing
>> >> >> Engine
>> >> >> >> + +-----------+ +---------------------------------+
>> >> >> >> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
>> >> >> >> + +-----------+ +---------------------------------+
>> >> >> >> + --------------------------------------------------------
>> >> >> >> + +-----------------------------------------------+
>> >> >> >> + | Zookeeper |
>> >> Distributed
>> >> >> >> Locking Service
>> >> >> >> + +-----------------------------------------------+
>> >> >> >> + --------------------------------------------------------
>> >> >> >> + +-----------------------------------------------+
>> >> >> >> + | |
>> >> >> >> + | +-----------------------------+ |
>> >> >> >> + | | HBase : Distributed Storage | |
>> >> >> >> + | +-----------------------------+ |
>> >> Distributed
>> >> >> >> Storage Systems
>> >> >> >> + | +---------------------------------+ |
>> >> >> >> + | | HDFS : Distributed File System | |
>> >> >> >> + | +---------------------------------+ |
>> >> >> >> + +-----------------------------------------------+
>> >> >> >> + }}}
>> >> >> >> +
>> >> >> >> == BSP ==
>> >> >> >>
>> >> >> >> The BSP package is a implementation of BSP over Hadoop
>> RPC(sockets).
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Best Regards, Edward J. Yoon
>> >> >> edwardyoon@apache.org
>> >> >> http://blog.udanax.org
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> edwardyoon@apache.org
>> >> http://blog.udanax.org
>> >>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>
>
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by Hyunsik Choi <hy...@gmail.com>.
> As mentioned above, I'd like to show only some example of how you
implement your own matrix/graph applications using BSP.
I'm not focusing on which one is better between BSP and M/R. I misunderstood
your intention due to the above your mention. It is because we will need not
only the BSP framework but also other programs to help your to develop
matrix algebra/graph processing.
Best regards,
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Tue, May 11, 2010 at 11:19 AM, Edward J. Yoon <ed...@apache.org>wrote:
> In this mailing
> (
> http://markmail.org/search/?q=hama-dev+discuss#query:hama-dev%20discuss+page:1+mid:amlvccbptom3yro3+state:results
> )
> and others, we discussed about:
>
> 1) We can't implement all of matrix algebra. 2) Even if we do that,
> the code complexity based on M/R model is too high. As we learned, the
> M/R model isn't fit well for the complex numerical/relational data
> processing required for many communications between each others. 3)
> Admittedly, BSP is flexible than M/R.
>
> Therefore, We've aligned our goal: to provide easy *matrix/graph
> computing* programming environment on the distributed system (hadoop),
> such as M/R framework. (NOT library) So, we won't develop only the BSP
> framework if there is other solutions.
>
> On Tue, May 11, 2010 at 10:29 AM, Hyunsik Choi <hy...@gmail.com>
> wrote:
> > According to you, the main objective of hama appears to be developing
> only
> > the BSP framework. Is it right?
> >
> > --
> > Hyunsik Choi
> > Database & Information Systems Group, Korea Univ.
> > http://diveintodata.org
> >
> >
> > On Mon, May 10, 2010 at 8:34 PM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >
> >> > In the distant future, the graph package would use both MR and BSP.
> But,
> >>
> >> Where will M/R use for?
> >>
> >> I'd like to cut M/R out of HAMA project and leave only the pure BSP &
> >> Graph computing framework as simple as possible. Then, the rest are
> >> user's matrix/graph applications.
> >>
> >> And, I'd like to consider about building an contributor/developer
> >> ECOsystem.
> >>
> >> > now i'll try to develop the graph package with only BSP. How about the
> >> > matrix package?
> >>
> >> As mentioned above, I'd like to show only some example of how you
> >> implement your own matrix/graph applications using BSP.
> >>
> >> I think the block of hadoop families should be added on diagram only
> >> to help understand relations between hadoop and hama,
> >>
> >> On Mon, May 10, 2010 at 7:40 PM, Hyunsik Choi <hy...@gmail.com>
> >> wrote:
> >> > In the distant future, the graph package would use both MR and BSP.
> But,
> >> > now i'll try to develop the graph package with only BSP. How about the
> >> > matrix package?
> >> >
> >> > --
> >> > Hyunsik Choi
> >> > Database & Information Systems Group, Korea Univ.
> >> > http://diveintodata.org
> >> >
> >> >
> >> > On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <
> edwardyoon@apache.org
> >> >wrote:
> >> >
> >> >> > zookeeper should be enclosed in BSP since zookeeper is not
> disclosed
> >> to
> >> >> > external. It is only used inside the BSP.
> >> >>
> >> >> Right you are. And also, the top of M/R and BSP should be clarified,
> >> >> but currently something is odd overall.
> >> >>
> >> >> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <
> hyunsik.choi@gmail.com>
> >> >> wrote:
> >> >> > Good job! But, the matrix package is omitted. Besides, I think that
> >> >> > zookeeper should be enclosed in BSP since zookeeper is not
> disclosed
> >> to
> >> >> > external. It is only used inside the BSP.
> >> >> >
> >> >> > Best regards,
> >> >> > --
> >> >> > Hyunsik Choi
> >> >> > Database & Information Systems Group, Korea Univ.
> >> >> > http://diveintodata.org
> >> >> >
> >> >> >
> >> >> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wikidiffs@apache.org
> >
> >> >> wrote:
> >> >> >
> >> >> >> Dear Wiki user,
> >> >> >>
> >> >> >> You have subscribed to a wiki page or wiki category on "Hama Wiki"
> >> for
> >> >> >> change notification.
> >> >> >>
> >> >> >> The "Architecture" page has been changed by Edward J. Yoon.
> >> >> >>
> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
> >> >> >>
> >> >> >> --------------------------------------------------
> >> >> >>
> >> >> >>
> >> >> >> This effort is still a "work in progress". Please feel free to
> add
> >> >> >> comments.
> >> >> >> ----
> >> >> >> +
> >> >> >> + == The overall architecture of HAMA ==
> >> >> >> + {{{
> >> >> >> + +-----------------------------------------------+
> >> >> >> + | HAMA APIs |
> >> >> >> + +-----------------------------------------------+
> >> >> >> + --------------------------------------------------------
> >> >> >> + +---------------------------+
> >> >> >> + | Graph Computing Framework |
> >> >> >> + +---------------------------+
> Computing
> >> >> Engine
> >> >> >> + +-----------+ +---------------------------------+
> >> >> >> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
> >> >> >> + +-----------+ +---------------------------------+
> >> >> >> + --------------------------------------------------------
> >> >> >> + +-----------------------------------------------+
> >> >> >> + | Zookeeper |
> >> Distributed
> >> >> >> Locking Service
> >> >> >> + +-----------------------------------------------+
> >> >> >> + --------------------------------------------------------
> >> >> >> + +-----------------------------------------------+
> >> >> >> + | |
> >> >> >> + | +-----------------------------+ |
> >> >> >> + | | HBase : Distributed Storage | |
> >> >> >> + | +-----------------------------+ |
> >> Distributed
> >> >> >> Storage Systems
> >> >> >> + | +---------------------------------+ |
> >> >> >> + | | HDFS : Distributed File System | |
> >> >> >> + | +---------------------------------+ |
> >> >> >> + +-----------------------------------------------+
> >> >> >> + }}}
> >> >> >> +
> >> >> >> == BSP ==
> >> >> >>
> >> >> >> The BSP package is a implementation of BSP over Hadoop
> RPC(sockets).
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best Regards, Edward J. Yoon
> >> >> edwardyoon@apache.org
> >> >> http://blog.udanax.org
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> edwardyoon@apache.org
> >> http://blog.udanax.org
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> edwardyoon@apache.org
> http://blog.udanax.org
>
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by "Edward J. Yoon" <ed...@apache.org>.
In this mailing
(http://markmail.org/search/?q=hama-dev+discuss#query:hama-dev%20discuss+page:1+mid:amlvccbptom3yro3+state:results)
and others, we discussed about:
1) We can't implement all of matrix algebra. 2) Even if we do that,
the code complexity based on M/R model is too high. As we learned, the
M/R model isn't fit well for the complex numerical/relational data
processing required for many communications between each others. 3)
Admittedly, BSP is flexible than M/R.
Therefore, We've aligned our goal: to provide easy *matrix/graph
computing* programming environment on the distributed system (hadoop),
such as M/R framework. (NOT library) So, we won't develop only the BSP
framework if there is other solutions.
On Tue, May 11, 2010 at 10:29 AM, Hyunsik Choi <hy...@gmail.com> wrote:
> According to you, the main objective of hama appears to be developing only
> the BSP framework. Is it right?
>
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
> On Mon, May 10, 2010 at 8:34 PM, Edward J. Yoon <ed...@apache.org>wrote:
>
>> > In the distant future, the graph package would use both MR and BSP. But,
>>
>> Where will M/R use for?
>>
>> I'd like to cut M/R out of HAMA project and leave only the pure BSP &
>> Graph computing framework as simple as possible. Then, the rest are
>> user's matrix/graph applications.
>>
>> And, I'd like to consider about building an contributor/developer
>> ECOsystem.
>>
>> > now i'll try to develop the graph package with only BSP. How about the
>> > matrix package?
>>
>> As mentioned above, I'd like to show only some example of how you
>> implement your own matrix/graph applications using BSP.
>>
>> I think the block of hadoop families should be added on diagram only
>> to help understand relations between hadoop and hama,
>>
>> On Mon, May 10, 2010 at 7:40 PM, Hyunsik Choi <hy...@gmail.com>
>> wrote:
>> > In the distant future, the graph package would use both MR and BSP. But,
>> > now i'll try to develop the graph package with only BSP. How about the
>> > matrix package?
>> >
>> > --
>> > Hyunsik Choi
>> > Database & Information Systems Group, Korea Univ.
>> > http://diveintodata.org
>> >
>> >
>> > On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <edwardyoon@apache.org
>> >wrote:
>> >
>> >> > zookeeper should be enclosed in BSP since zookeeper is not disclosed
>> to
>> >> > external. It is only used inside the BSP.
>> >>
>> >> Right you are. And also, the top of M/R and BSP should be clarified,
>> >> but currently something is odd overall.
>> >>
>> >> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <hy...@gmail.com>
>> >> wrote:
>> >> > Good job! But, the matrix package is omitted. Besides, I think that
>> >> > zookeeper should be enclosed in BSP since zookeeper is not disclosed
>> to
>> >> > external. It is only used inside the BSP.
>> >> >
>> >> > Best regards,
>> >> > --
>> >> > Hyunsik Choi
>> >> > Database & Information Systems Group, Korea Univ.
>> >> > http://diveintodata.org
>> >> >
>> >> >
>> >> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wi...@apache.org>
>> >> wrote:
>> >> >
>> >> >> Dear Wiki user,
>> >> >>
>> >> >> You have subscribed to a wiki page or wiki category on "Hama Wiki"
>> for
>> >> >> change notification.
>> >> >>
>> >> >> The "Architecture" page has been changed by Edward J. Yoon.
>> >> >> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
>> >> >>
>> >> >> --------------------------------------------------
>> >> >>
>> >> >>
>> >> >> This effort is still a "work in progress". Please feel free to add
>> >> >> comments.
>> >> >> ----
>> >> >> +
>> >> >> + == The overall architecture of HAMA ==
>> >> >> + {{{
>> >> >> + +-----------------------------------------------+
>> >> >> + | HAMA APIs |
>> >> >> + +-----------------------------------------------+
>> >> >> + --------------------------------------------------------
>> >> >> + +---------------------------+
>> >> >> + | Graph Computing Framework |
>> >> >> + +---------------------------+ Computing
>> >> Engine
>> >> >> + +-----------+ +---------------------------------+
>> >> >> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
>> >> >> + +-----------+ +---------------------------------+
>> >> >> + --------------------------------------------------------
>> >> >> + +-----------------------------------------------+
>> >> >> + | Zookeeper |
>> Distributed
>> >> >> Locking Service
>> >> >> + +-----------------------------------------------+
>> >> >> + --------------------------------------------------------
>> >> >> + +-----------------------------------------------+
>> >> >> + | |
>> >> >> + | +-----------------------------+ |
>> >> >> + | | HBase : Distributed Storage | |
>> >> >> + | +-----------------------------+ |
>> Distributed
>> >> >> Storage Systems
>> >> >> + | +---------------------------------+ |
>> >> >> + | | HDFS : Distributed File System | |
>> >> >> + | +---------------------------------+ |
>> >> >> + +-----------------------------------------------+
>> >> >> + }}}
>> >> >> +
>> >> >> == BSP ==
>> >> >>
>> >> >> The BSP package is a implementation of BSP over Hadoop RPC(sockets).
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> edwardyoon@apache.org
>> >> http://blog.udanax.org
>> >>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>
--
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by Hyunsik Choi <hy...@gmail.com>.
According to you, the main objective of hama appears to be developing only
the BSP framework. Is it right?
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Mon, May 10, 2010 at 8:34 PM, Edward J. Yoon <ed...@apache.org>wrote:
> > In the distant future, the graph package would use both MR and BSP. But,
>
> Where will M/R use for?
>
> I'd like to cut M/R out of HAMA project and leave only the pure BSP &
> Graph computing framework as simple as possible. Then, the rest are
> user's matrix/graph applications.
>
> And, I'd like to consider about building an contributor/developer
> ECOsystem.
>
> > now i'll try to develop the graph package with only BSP. How about the
> > matrix package?
>
> As mentioned above, I'd like to show only some example of how you
> implement your own matrix/graph applications using BSP.
>
> I think the block of hadoop families should be added on diagram only
> to help understand relations between hadoop and hama,
>
> On Mon, May 10, 2010 at 7:40 PM, Hyunsik Choi <hy...@gmail.com>
> wrote:
> > In the distant future, the graph package would use both MR and BSP. But,
> > now i'll try to develop the graph package with only BSP. How about the
> > matrix package?
> >
> > --
> > Hyunsik Choi
> > Database & Information Systems Group, Korea Univ.
> > http://diveintodata.org
> >
> >
> > On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >
> >> > zookeeper should be enclosed in BSP since zookeeper is not disclosed
> to
> >> > external. It is only used inside the BSP.
> >>
> >> Right you are. And also, the top of M/R and BSP should be clarified,
> >> but currently something is odd overall.
> >>
> >> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <hy...@gmail.com>
> >> wrote:
> >> > Good job! But, the matrix package is omitted. Besides, I think that
> >> > zookeeper should be enclosed in BSP since zookeeper is not disclosed
> to
> >> > external. It is only used inside the BSP.
> >> >
> >> > Best regards,
> >> > --
> >> > Hyunsik Choi
> >> > Database & Information Systems Group, Korea Univ.
> >> > http://diveintodata.org
> >> >
> >> >
> >> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wi...@apache.org>
> >> wrote:
> >> >
> >> >> Dear Wiki user,
> >> >>
> >> >> You have subscribed to a wiki page or wiki category on "Hama Wiki"
> for
> >> >> change notification.
> >> >>
> >> >> The "Architecture" page has been changed by Edward J. Yoon.
> >> >> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
> >> >>
> >> >> --------------------------------------------------
> >> >>
> >> >>
> >> >> This effort is still a "work in progress". Please feel free to add
> >> >> comments.
> >> >> ----
> >> >> +
> >> >> + == The overall architecture of HAMA ==
> >> >> + {{{
> >> >> + +-----------------------------------------------+
> >> >> + | HAMA APIs |
> >> >> + +-----------------------------------------------+
> >> >> + --------------------------------------------------------
> >> >> + +---------------------------+
> >> >> + | Graph Computing Framework |
> >> >> + +---------------------------+ Computing
> >> Engine
> >> >> + +-----------+ +---------------------------------+
> >> >> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
> >> >> + +-----------+ +---------------------------------+
> >> >> + --------------------------------------------------------
> >> >> + +-----------------------------------------------+
> >> >> + | Zookeeper |
> Distributed
> >> >> Locking Service
> >> >> + +-----------------------------------------------+
> >> >> + --------------------------------------------------------
> >> >> + +-----------------------------------------------+
> >> >> + | |
> >> >> + | +-----------------------------+ |
> >> >> + | | HBase : Distributed Storage | |
> >> >> + | +-----------------------------+ |
> Distributed
> >> >> Storage Systems
> >> >> + | +---------------------------------+ |
> >> >> + | | HDFS : Distributed File System | |
> >> >> + | +---------------------------------+ |
> >> >> + +-----------------------------------------------+
> >> >> + }}}
> >> >> +
> >> >> == BSP ==
> >> >>
> >> >> The BSP package is a implementation of BSP over Hadoop RPC(sockets).
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> edwardyoon@apache.org
> >> http://blog.udanax.org
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> edwardyoon@apache.org
> http://blog.udanax.org
>
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by "Edward J. Yoon" <ed...@apache.org>.
> In the distant future, the graph package would use both MR and BSP. But,
Where will M/R use for?
I'd like to cut M/R out of HAMA project and leave only the pure BSP &
Graph computing framework as simple as possible. Then, the rest are
user's matrix/graph applications.
And, I'd like to consider about building an contributor/developer ECOsystem.
> now i'll try to develop the graph package with only BSP. How about the
> matrix package?
As mentioned above, I'd like to show only some example of how you
implement your own matrix/graph applications using BSP.
I think the block of hadoop families should be added on diagram only
to help understand relations between hadoop and hama,
On Mon, May 10, 2010 at 7:40 PM, Hyunsik Choi <hy...@gmail.com> wrote:
> In the distant future, the graph package would use both MR and BSP. But,
> now i'll try to develop the graph package with only BSP. How about the
> matrix package?
>
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
> On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <ed...@apache.org>wrote:
>
>> > zookeeper should be enclosed in BSP since zookeeper is not disclosed to
>> > external. It is only used inside the BSP.
>>
>> Right you are. And also, the top of M/R and BSP should be clarified,
>> but currently something is odd overall.
>>
>> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <hy...@gmail.com>
>> wrote:
>> > Good job! But, the matrix package is omitted. Besides, I think that
>> > zookeeper should be enclosed in BSP since zookeeper is not disclosed to
>> > external. It is only used inside the BSP.
>> >
>> > Best regards,
>> > --
>> > Hyunsik Choi
>> > Database & Information Systems Group, Korea Univ.
>> > http://diveintodata.org
>> >
>> >
>> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wi...@apache.org>
>> wrote:
>> >
>> >> Dear Wiki user,
>> >>
>> >> You have subscribed to a wiki page or wiki category on "Hama Wiki" for
>> >> change notification.
>> >>
>> >> The "Architecture" page has been changed by Edward J. Yoon.
>> >> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
>> >>
>> >> --------------------------------------------------
>> >>
>> >>
>> >> This effort is still a "work in progress". Please feel free to add
>> >> comments.
>> >> ----
>> >> +
>> >> + == The overall architecture of HAMA ==
>> >> + {{{
>> >> + +-----------------------------------------------+
>> >> + | HAMA APIs |
>> >> + +-----------------------------------------------+
>> >> + --------------------------------------------------------
>> >> + +---------------------------+
>> >> + | Graph Computing Framework |
>> >> + +---------------------------+ Computing
>> Engine
>> >> + +-----------+ +---------------------------------+
>> >> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
>> >> + +-----------+ +---------------------------------+
>> >> + --------------------------------------------------------
>> >> + +-----------------------------------------------+
>> >> + | Zookeeper | Distributed
>> >> Locking Service
>> >> + +-----------------------------------------------+
>> >> + --------------------------------------------------------
>> >> + +-----------------------------------------------+
>> >> + | |
>> >> + | +-----------------------------+ |
>> >> + | | HBase : Distributed Storage | |
>> >> + | +-----------------------------+ | Distributed
>> >> Storage Systems
>> >> + | +---------------------------------+ |
>> >> + | | HDFS : Distributed File System | |
>> >> + | +---------------------------------+ |
>> >> + +-----------------------------------------------+
>> >> + }}}
>> >> +
>> >> == BSP ==
>> >>
>> >> The BSP package is a implementation of BSP over Hadoop RPC(sockets).
>> >>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>
--
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by Hyunsik Choi <hy...@gmail.com>.
In the distant future, the graph package would use both MR and BSP. But,
now i'll try to develop the graph package with only BSP. How about the
matrix package?
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <ed...@apache.org>wrote:
> > zookeeper should be enclosed in BSP since zookeeper is not disclosed to
> > external. It is only used inside the BSP.
>
> Right you are. And also, the top of M/R and BSP should be clarified,
> but currently something is odd overall.
>
> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <hy...@gmail.com>
> wrote:
> > Good job! But, the matrix package is omitted. Besides, I think that
> > zookeeper should be enclosed in BSP since zookeeper is not disclosed to
> > external. It is only used inside the BSP.
> >
> > Best regards,
> > --
> > Hyunsik Choi
> > Database & Information Systems Group, Korea Univ.
> > http://diveintodata.org
> >
> >
> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wi...@apache.org>
> wrote:
> >
> >> Dear Wiki user,
> >>
> >> You have subscribed to a wiki page or wiki category on "Hama Wiki" for
> >> change notification.
> >>
> >> The "Architecture" page has been changed by Edward J. Yoon.
> >> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
> >>
> >> --------------------------------------------------
> >>
> >>
> >> This effort is still a "work in progress". Please feel free to add
> >> comments.
> >> ----
> >> +
> >> + == The overall architecture of HAMA ==
> >> + {{{
> >> + +-----------------------------------------------+
> >> + | HAMA APIs |
> >> + +-----------------------------------------------+
> >> + --------------------------------------------------------
> >> + +---------------------------+
> >> + | Graph Computing Framework |
> >> + +---------------------------+ Computing
> Engine
> >> + +-----------+ +---------------------------------+
> >> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
> >> + +-----------+ +---------------------------------+
> >> + --------------------------------------------------------
> >> + +-----------------------------------------------+
> >> + | Zookeeper | Distributed
> >> Locking Service
> >> + +-----------------------------------------------+
> >> + --------------------------------------------------------
> >> + +-----------------------------------------------+
> >> + | |
> >> + | +-----------------------------+ |
> >> + | | HBase : Distributed Storage | |
> >> + | +-----------------------------+ | Distributed
> >> Storage Systems
> >> + | +---------------------------------+ |
> >> + | | HDFS : Distributed File System | |
> >> + | +---------------------------------+ |
> >> + +-----------------------------------------------+
> >> + }}}
> >> +
> >> == BSP ==
> >>
> >> The BSP package is a implementation of BSP over Hadoop RPC(sockets).
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> edwardyoon@apache.org
> http://blog.udanax.org
>
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by "Edward J. Yoon" <ed...@apache.org>.
> zookeeper should be enclosed in BSP since zookeeper is not disclosed to
> external. It is only used inside the BSP.
Right you are. And also, the top of M/R and BSP should be clarified,
but currently something is odd overall.
On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <hy...@gmail.com> wrote:
> Good job! But, the matrix package is omitted. Besides, I think that
> zookeeper should be enclosed in BSP since zookeeper is not disclosed to
> external. It is only used inside the BSP.
>
> Best regards,
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
> On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wi...@apache.org> wrote:
>
>> Dear Wiki user,
>>
>> You have subscribed to a wiki page or wiki category on "Hama Wiki" for
>> change notification.
>>
>> The "Architecture" page has been changed by Edward J. Yoon.
>> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
>>
>> --------------------------------------------------
>>
>>
>> This effort is still a "work in progress". Please feel free to add
>> comments.
>> ----
>> +
>> + == The overall architecture of HAMA ==
>> + {{{
>> + +-----------------------------------------------+
>> + | HAMA APIs |
>> + +-----------------------------------------------+
>> + --------------------------------------------------------
>> + +---------------------------+
>> + | Graph Computing Framework |
>> + +---------------------------+ Computing Engine
>> + +-----------+ +---------------------------------+
>> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
>> + +-----------+ +---------------------------------+
>> + --------------------------------------------------------
>> + +-----------------------------------------------+
>> + | Zookeeper | Distributed
>> Locking Service
>> + +-----------------------------------------------+
>> + --------------------------------------------------------
>> + +-----------------------------------------------+
>> + | |
>> + | +-----------------------------+ |
>> + | | HBase : Distributed Storage | |
>> + | +-----------------------------+ | Distributed
>> Storage Systems
>> + | +---------------------------------+ |
>> + | | HDFS : Distributed File System | |
>> + | +---------------------------------+ |
>> + +-----------------------------------------------+
>> + }}}
>> +
>> == BSP ==
>>
>> The BSP package is a implementation of BSP over Hadoop RPC(sockets).
>>
>
--
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org
Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Posted by Hyunsik Choi <hy...@gmail.com>.
Good job! But, the matrix package is omitted. Besides, I think that
zookeeper should be enclosed in BSP since zookeeper is not disclosed to
external. It is only used inside the BSP.
Best regards,
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wi...@apache.org> wrote:
> Dear Wiki user,
>
> You have subscribed to a wiki page or wiki category on "Hama Wiki" for
> change notification.
>
> The "Architecture" page has been changed by Edward J. Yoon.
> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
>
> --------------------------------------------------
>
>
> This effort is still a "work in progress". Please feel free to add
> comments.
> ----
> +
> + == The overall architecture of HAMA ==
> + {{{
> + +-----------------------------------------------+
> + | HAMA APIs |
> + +-----------------------------------------------+
> + --------------------------------------------------------
> + +---------------------------+
> + | Graph Computing Framework |
> + +---------------------------+ Computing Engine
> + +-----------+ +---------------------------------+
> + | MapReduce | | BSP (Bulk Synchronous Parallel) |
> + +-----------+ +---------------------------------+
> + --------------------------------------------------------
> + +-----------------------------------------------+
> + | Zookeeper | Distributed
> Locking Service
> + +-----------------------------------------------+
> + --------------------------------------------------------
> + +-----------------------------------------------+
> + | |
> + | +-----------------------------+ |
> + | | HBase : Distributed Storage | |
> + | +-----------------------------+ | Distributed
> Storage Systems
> + | +---------------------------------+ |
> + | | HDFS : Distributed File System | |
> + | +---------------------------------+ |
> + +-----------------------------------------------+
> + }}}
> +
> == BSP ==
>
> The BSP package is a implementation of BSP over Hadoop RPC(sockets).
>