You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by Sangwon Seo <sm...@gmail.com> on 2009/12/11 08:55:05 UTC
RE: Please review a document about the BSP package
Cool~!
Are there any differences with traditional BSP?
For example (as expected from others), Map Reduce also has good advantages,
so if application is not fit with MR, It's better to use other method such as MPI/OpenMP and one of other BSP packages.
That is, is there something meaningful of comparison between BSP and MR ?
(This question is also from prof.Kim before)
I mean why don't you add this answer in slides.
Thanks.
Best Regards,
Sangwon Seo.
Sangwon Seo
Computer Architecture Lab,
Computer Science Department, School of EECS,
KAIST (Korea Advanced Institute of Science and Technology), Republic of Korea.
http://smiler-note.blogspot.com
-----Original Message-----
From: Edward J. Yoon [mailto:edwardyoon@apache.org]
Sent: Friday, December 11, 2009 11:31 PM
To: hama-dev@incubator.apache.org
Subject: Please review a document about the BSP package
Hello all,
I've worked on documenting the BSP package with Hyunsick Choi --
http://docs.google.com/present/view?id=dc8qtchr_1f44rjccd
Any advices are welcome.
--
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org
RE: Please review a document about the BSP package
Posted by Sangwon Seo <sm...@gmail.com>.
Dear Hyunsik Choi,
Yes very good!
And I reasonably understood why we adapt BSP as a implementation view point.
However, I and Edward.J Yoon are now preparing the HAMA paper for upcoming
USENX workshop (only deal with matrix).
It soon will introduce for us.
Evaluation is working on high speed SCI-Cluster (16 nodes, 2D Torus
Topology).
This cluster somewhat overcomes locality problem and communication
(shuffling) overhead with very low latency network connection.
As your words, my point of view was for academic than implementation. ^^
I want to make academic issues clear as well as implementation, even if BSP
is very old idea in research.
If I have some idea, I will suggest about it.
I believe that it will upward Apache HAMA as a top project of ASF.
Anyway, BSP package is good itself!!!
-----Original Message-----
From: Hyunsik Choi [mailto:hyunsik.choi@gmail.com]
Sent: Saturday, December 12, 2009 9:51 PM
To: hama-dev
Subject: Re: Please review a document about the BSP package
Hi Sangwon,
I welcome to any debate because intensive arguments generally make our
ideas more robust :)
Anyway, the main objective of BSP package is not for academic. We do
not necessarily need to make a difference from the traditional BSP.
First of all, we aims at making BSP available on Hadoop since now we
do not have any other choice except for MapReduce and BSP can help to
develop other types of applications that is not appropriate to
MapReduce. In addition, now the traditional BSP does not fully
satisfy the direction (i.e., straggling mechanism and fault tolerance)
that I mentioned in the previous post. Thus, we are trying to adapt
BSP to the characteristics of analytical processing and cloud
computing environments.
Best wishes,
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Sat, Dec 12, 2009 at 11:30 AM, Sangwon Seo <sm...@gmail.com> wrote:
> I agree with your good mentions.
>
> (This argue is for our team. ^^)
> However,
> In cloud computing environments,
> It goes direction closing to immutable and share-noting architecture.
> However two thing you mentioned (i,ii) are also significantly considered
in
> Hadoop MR.
> For example, straggler finding mechanism for heterogonous system.
> I think we need more specific reasons of HAMA BSP.
> I hope our introduced framework base on BSP has good advantages though
> exploiting the differentiality with legacy BSP and will-convergence with
> Hadoop.
>
> Thanks.
>
> Best Regards,
> Sangwon Seo.
>
> -----Original Message-----
> From: Hyunsik Choi [mailto:hyunsik.choi@gmail.com]
> Sent: Saturday, December 12, 2009 11:40 AM
> To: hama-dev
> Subject: Re: Please review a document about the BSP package
>
> Hi,
>
> HAMA aims at a scientific package on cloud computing environment. As
> you know, cloud computing environment is not business or trends
> keyword. It is inevitable movement because cloud computing provides
> business agility and it is very important to some kinds of companies.
> Many companies already are using this environment. Acutually, now our
> BSP still is not different to traditional BSP. However, the BSP on
> Hadoop is an trial to develop another distributed computing model on
> cloud computing environment. In addition, we know that our BSP has to
> go which direction.
>
> In cloud computing environment, computing resource is readily
> allocated without interference. Thus, cloud computing environment
> needs to be based on shared-nothing architecture for easy assigning
> independent computing resource connected in network.
>
> In shared-nothing architecture, transactional operations that execute
> frequent read and write are not appropriate since it is hard to
> achieve requirements for transaction. Consequently, data processing
> should be analytical processing (mostly read processing) in this
> environment.
>
> In order to develop some design in this environment, we have to
> consider some important characteristics of cloud computing and
> analytical processing.
>
> i) Analytical processing usually takes several hours or days.
>
> Therefore, we have to consider fault tolerance if we don't want to
> restart long time job.
>
> ii) The capabilities of cluster nodes on cloud may not be even.
>
> In other words, their hardware specifications may be different to each
> other. In this situation, the processing will be continuous until jobs
> on the slowest computer instance are terminated. MR has a mechanism
> for this nature.
>
> Above characteristics have to be considered in cloud computing
> environment. Now, our BSP is still not good at above characteristic
> and is still too simple framework. However, we have some ideas for
> above characteristics. Later, we try to achieve above features and
> improve further things.
>
> Best regards,
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
>
> On Fri, Dec 11, 2009 at 4:55 PM, Sangwon Seo <sm...@gmail.com> wrote:
>> Cool~!
>> Are there any differences with traditional BSP?
>> For example (as expected from others), Map Reduce also has good
> advantages,
>> so if application is not fit with MR, It's better to use other method
> such as MPI/OpenMP and one of other BSP packages.
>> That is, is there something meaningful of comparison between BSP and MR ?
>> (This question is also from prof.Kim before)
>> I mean why don't you add this answer in slides.
>>
>> Thanks.
>>
>> Best Regards,
>> Sangwon Seo.
>>
>> Sangwon Seo
>> Computer Architecture Lab,
>> Computer Science Department, School of EECS,
>> KAIST (Korea Advanced Institute of Science and Technology), Republic of
> Korea.
>> http://smiler-note.blogspot.com
>>
>>
>>
>> -----Original Message-----
>> From: Edward J. Yoon [mailto:edwardyoon@apache.org]
>> Sent: Friday, December 11, 2009 11:31 PM
>> To: hama-dev@incubator.apache.org
>> Subject: Please review a document about the BSP package
>>
>> Hello all,
>>
>> I've worked on documenting the BSP package with Hyunsick Choi --
>> http://docs.google.com/present/view?id=dc8qtchr_1f44rjccd
>>
>> Any advices are welcome.
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>>
>
>
Re: Please review a document about the BSP package
Posted by Hyunsik Choi <hy...@gmail.com>.
Hi Sangwon,
I welcome to any debate because intensive arguments generally make our
ideas more robust :)
Anyway, the main objective of BSP package is not for academic. We do
not necessarily need to make a difference from the traditional BSP.
First of all, we aims at making BSP available on Hadoop since now we
do not have any other choice except for MapReduce and BSP can help to
develop other types of applications that is not appropriate to
MapReduce. In addition, now the traditional BSP does not fully
satisfy the direction (i.e., straggling mechanism and fault tolerance)
that I mentioned in the previous post. Thus, we are trying to adapt
BSP to the characteristics of analytical processing and cloud
computing environments.
Best wishes,
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Sat, Dec 12, 2009 at 11:30 AM, Sangwon Seo <sm...@gmail.com> wrote:
> I agree with your good mentions.
>
> (This argue is for our team. ^^)
> However,
> In cloud computing environments,
> It goes direction closing to immutable and share-noting architecture.
> However two thing you mentioned (i,ii) are also significantly considered in
> Hadoop MR.
> For example, straggler finding mechanism for heterogonous system.
> I think we need more specific reasons of HAMA BSP.
> I hope our introduced framework base on BSP has good advantages though
> exploiting the differentiality with legacy BSP and will-convergence with
> Hadoop.
>
> Thanks.
>
> Best Regards,
> Sangwon Seo.
>
> -----Original Message-----
> From: Hyunsik Choi [mailto:hyunsik.choi@gmail.com]
> Sent: Saturday, December 12, 2009 11:40 AM
> To: hama-dev
> Subject: Re: Please review a document about the BSP package
>
> Hi,
>
> HAMA aims at a scientific package on cloud computing environment. As
> you know, cloud computing environment is not business or trends
> keyword. It is inevitable movement because cloud computing provides
> business agility and it is very important to some kinds of companies.
> Many companies already are using this environment. Acutually, now our
> BSP still is not different to traditional BSP. However, the BSP on
> Hadoop is an trial to develop another distributed computing model on
> cloud computing environment. In addition, we know that our BSP has to
> go which direction.
>
> In cloud computing environment, computing resource is readily
> allocated without interference. Thus, cloud computing environment
> needs to be based on shared-nothing architecture for easy assigning
> independent computing resource connected in network.
>
> In shared-nothing architecture, transactional operations that execute
> frequent read and write are not appropriate since it is hard to
> achieve requirements for transaction. Consequently, data processing
> should be analytical processing (mostly read processing) in this
> environment.
>
> In order to develop some design in this environment, we have to
> consider some important characteristics of cloud computing and
> analytical processing.
>
> i) Analytical processing usually takes several hours or days.
>
> Therefore, we have to consider fault tolerance if we don't want to
> restart long time job.
>
> ii) The capabilities of cluster nodes on cloud may not be even.
>
> In other words, their hardware specifications may be different to each
> other. In this situation, the processing will be continuous until jobs
> on the slowest computer instance are terminated. MR has a mechanism
> for this nature.
>
> Above characteristics have to be considered in cloud computing
> environment. Now, our BSP is still not good at above characteristic
> and is still too simple framework. However, we have some ideas for
> above characteristics. Later, we try to achieve above features and
> improve further things.
>
> Best regards,
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
>
> On Fri, Dec 11, 2009 at 4:55 PM, Sangwon Seo <sm...@gmail.com> wrote:
>> Cool~!
>> Are there any differences with traditional BSP?
>> For example (as expected from others), Map Reduce also has good
> advantages,
>> so if application is not fit with MR, It's better to use other method
> such as MPI/OpenMP and one of other BSP packages.
>> That is, is there something meaningful of comparison between BSP and MR ?
>> (This question is also from prof.Kim before)
>> I mean why don't you add this answer in slides.
>>
>> Thanks.
>>
>> Best Regards,
>> Sangwon Seo.
>>
>> Sangwon Seo
>> Computer Architecture Lab,
>> Computer Science Department, School of EECS,
>> KAIST (Korea Advanced Institute of Science and Technology), Republic of
> Korea.
>> http://smiler-note.blogspot.com
>>
>>
>>
>> -----Original Message-----
>> From: Edward J. Yoon [mailto:edwardyoon@apache.org]
>> Sent: Friday, December 11, 2009 11:31 PM
>> To: hama-dev@incubator.apache.org
>> Subject: Please review a document about the BSP package
>>
>> Hello all,
>>
>> I've worked on documenting the BSP package with Hyunsick Choi --
>> http://docs.google.com/present/view?id=dc8qtchr_1f44rjccd
>>
>> Any advices are welcome.
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>>
>
>
RE: Please review a document about the BSP package
Posted by Sangwon Seo <sm...@gmail.com>.
I agree with your good mentions.
(This argue is for our team. ^^)
However,
In cloud computing environments,
It goes direction closing to immutable and share-noting architecture.
However two thing you mentioned (i,ii) are also significantly considered in
Hadoop MR.
For example, straggler finding mechanism for heterogonous system.
I think we need more specific reasons of HAMA BSP.
I hope our introduced framework base on BSP has good advantages though
exploiting the differentiality with legacy BSP and will-convergence with
Hadoop.
Thanks.
Best Regards,
Sangwon Seo.
-----Original Message-----
From: Hyunsik Choi [mailto:hyunsik.choi@gmail.com]
Sent: Saturday, December 12, 2009 11:40 AM
To: hama-dev
Subject: Re: Please review a document about the BSP package
Hi,
HAMA aims at a scientific package on cloud computing environment. As
you know, cloud computing environment is not business or trends
keyword. It is inevitable movement because cloud computing provides
business agility and it is very important to some kinds of companies.
Many companies already are using this environment. Acutually, now our
BSP still is not different to traditional BSP. However, the BSP on
Hadoop is an trial to develop another distributed computing model on
cloud computing environment. In addition, we know that our BSP has to
go which direction.
In cloud computing environment, computing resource is readily
allocated without interference. Thus, cloud computing environment
needs to be based on shared-nothing architecture for easy assigning
independent computing resource connected in network.
In shared-nothing architecture, transactional operations that execute
frequent read and write are not appropriate since it is hard to
achieve requirements for transaction. Consequently, data processing
should be analytical processing (mostly read processing) in this
environment.
In order to develop some design in this environment, we have to
consider some important characteristics of cloud computing and
analytical processing.
i) Analytical processing usually takes several hours or days.
Therefore, we have to consider fault tolerance if we don't want to
restart long time job.
ii) The capabilities of cluster nodes on cloud may not be even.
In other words, their hardware specifications may be different to each
other. In this situation, the processing will be continuous until jobs
on the slowest computer instance are terminated. MR has a mechanism
for this nature.
Above characteristics have to be considered in cloud computing
environment. Now, our BSP is still not good at above characteristic
and is still too simple framework. However, we have some ideas for
above characteristics. Later, we try to achieve above features and
improve further things.
Best regards,
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Fri, Dec 11, 2009 at 4:55 PM, Sangwon Seo <sm...@gmail.com> wrote:
> Cool~!
> Are there any differences with traditional BSP?
> For example (as expected from others), Map Reduce also has good
advantages,
> so if application is not fit with MR, It's better to use other method
such as MPI/OpenMP and one of other BSP packages.
> That is, is there something meaningful of comparison between BSP and MR ?
> (This question is also from prof.Kim before)
> I mean why don't you add this answer in slides.
>
> Thanks.
>
> Best Regards,
> Sangwon Seo.
>
> Sangwon Seo
> Computer Architecture Lab,
> Computer Science Department, School of EECS,
> KAIST (Korea Advanced Institute of Science and Technology), Republic of
Korea.
> http://smiler-note.blogspot.com
>
>
>
> -----Original Message-----
> From: Edward J. Yoon [mailto:edwardyoon@apache.org]
> Sent: Friday, December 11, 2009 11:31 PM
> To: hama-dev@incubator.apache.org
> Subject: Please review a document about the BSP package
>
> Hello all,
>
> I've worked on documenting the BSP package with Hyunsick Choi --
> http://docs.google.com/present/view?id=dc8qtchr_1f44rjccd
>
> Any advices are welcome.
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> edwardyoon@apache.org
> http://blog.udanax.org
>
>
Re: Please review a document about the BSP package
Posted by Hyunsik Choi <hy...@gmail.com>.
Hi,
HAMA aims at a scientific package on cloud computing environment. As
you know, cloud computing environment is not business or trends
keyword. It is inevitable movement because cloud computing provides
business agility and it is very important to some kinds of companies.
Many companies already are using this environment. Acutually, now our
BSP still is not different to traditional BSP. However, the BSP on
Hadoop is an trial to develop another distributed computing model on
cloud computing environment. In addition, we know that our BSP has to
go which direction.
In cloud computing environment, computing resource is readily
allocated without interference. Thus, cloud computing environment
needs to be based on shared-nothing architecture for easy assigning
independent computing resource connected in network.
In shared-nothing architecture, transactional operations that execute
frequent read and write are not appropriate since it is hard to
achieve requirements for transaction. Consequently, data processing
should be analytical processing (mostly read processing) in this
environment.
In order to develop some design in this environment, we have to
consider some important characteristics of cloud computing and
analytical processing.
i) Analytical processing usually takes several hours or days.
Therefore, we have to consider fault tolerance if we don't want to
restart long time job.
ii) The capabilities of cluster nodes on cloud may not be even.
In other words, their hardware specifications may be different to each
other. In this situation, the processing will be continuous until jobs
on the slowest computer instance are terminated. MR has a mechanism
for this nature.
Above characteristics have to be considered in cloud computing
environment. Now, our BSP is still not good at above characteristic
and is still too simple framework. However, we have some ideas for
above characteristics. Later, we try to achieve above features and
improve further things.
Best regards,
--
Hyunsik Choi
Database & Information Systems Group, Korea Univ.
http://diveintodata.org
On Fri, Dec 11, 2009 at 4:55 PM, Sangwon Seo <sm...@gmail.com> wrote:
> Cool~!
> Are there any differences with traditional BSP?
> For example (as expected from others), Map Reduce also has good advantages,
> so if application is not fit with MR, It's better to use other method such as MPI/OpenMP and one of other BSP packages.
> That is, is there something meaningful of comparison between BSP and MR ?
> (This question is also from prof.Kim before)
> I mean why don't you add this answer in slides.
>
> Thanks.
>
> Best Regards,
> Sangwon Seo.
>
> Sangwon Seo
> Computer Architecture Lab,
> Computer Science Department, School of EECS,
> KAIST (Korea Advanced Institute of Science and Technology), Republic of Korea.
> http://smiler-note.blogspot.com
>
>
>
> -----Original Message-----
> From: Edward J. Yoon [mailto:edwardyoon@apache.org]
> Sent: Friday, December 11, 2009 11:31 PM
> To: hama-dev@incubator.apache.org
> Subject: Please review a document about the BSP package
>
> Hello all,
>
> I've worked on documenting the BSP package with Hyunsick Choi --
> http://docs.google.com/present/view?id=dc8qtchr_1f44rjccd
>
> Any advices are welcome.
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> edwardyoon@apache.org
> http://blog.udanax.org
>
>