You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Jan van der Lugt <ja...@oracle.com> on 2012/07/11 03:25:06 UTC
Announcing: Green-Marl/Giraph
Hi everyone,
Some of you might already know Green-Marl, Green-Marl compatibility was
even mentioned as a new feature for Giraph 0.2, but for those who don't,
let me give you a quick introduction.
Green-Marl is a domain-specific language tailored to graph algorithms. It
has many features that makes it possible to write graph algorithms very
concise and intuitive. For example, it has built-in constructs for graphs,
node properties, and traversals. A more complete description is given here:
http://ppl.stanford.edu/papers/asplos12_hong.pdf. Apart from the language,
there is also a Green-Marl compiler that compiles the language to different
targets. C++/OpenMP was initially supported, after which support for
Stanford GPS (a Pregel-clone developed at Stanford) was added for a subset
of the features in the Green-Marl language. The big advantage of using a
language like Green-Marl is that it enables you to write implicitly
parallel programs in an intuitive way with the compiler doing most of the
heavy work such as generating the messaging, converting pull-based
operation (remote reading) to push-based operations (message sending), etc.
In the last few months I have worked on a few features in Giraph that
enable Green-Marl to target Giraph as a back-end (namely GIRAPH-127,
GIRAPH-192 and GIRAPH-216). Since these have all been merged, Green-Marl
and Giraph should be compatible as of last week!
Using Green-Marl with Giraph is very simple, as long as you have a working
Hadoop/Giraph environment. Since the Green-Marl compiler is a
source-to-source compiler, it is very easy to use it in an existing set-up.
The steps to using Green-Marl are like this:
1. Download the Green-Marl compiler from
https://github.com/stanford-ppl/Green-Marl, run make_dirs.sh in the
top-level dir and run make in the src dir. A more detailed explanation is
given in the Github readme.
2. Compile one of the examples (in the top-level directory) like
this: bin/gm_comp -t=giraph apps/src/pagerank.gm
3. Move pagerank.java to the Giraph src directory or your own project,
package it into a .jar and go!
4. Explore more examples, modify them and start writing your own
Green-Marl is actively being worked on within Oracle Labs, new features are
being added to the language (such as collection node properties) and
support for these features will be added to the distributed back-ends in
the near future.
In case of questions or suggestions, please send us a message! Feedback is
very much appreciated and people are more than welcome to contribute
everything from small bugfixes to entire back-ends ;-)
- Jan
Re: Announcing: Green-Marl/Giraph
Posted by Eli Reisman <in...@gmail.com>.
I will be attending a lecture from some of the Stanford folks on this soon,
will let you know how it goes. This development can aid adoption of GIraph
more widely!
On Wed, Jul 11, 2012 at 12:57 PM, Claudio Martella <
claudio.martella@gmail.com> wrote:
> This is really really cool stuff. congratulations!
>
> On Wed, Jul 11, 2012 at 3:25 AM, Jan van der Lugt
> <ja...@oracle.com> wrote:
> > Hi everyone,
> >
> > Some of you might already know Green-Marl, Green-Marl compatibility was
> > even mentioned as a new feature for Giraph 0.2, but for those who don't,
> > let me give you a quick introduction.
> >
> > Green-Marl is a domain-specific language tailored to graph algorithms. It
> > has many features that makes it possible to write graph algorithms very
> > concise and intuitive. For example, it has built-in constructs for
> graphs,
> > node properties, and traversals. A more complete description is given
> here:
> > http://ppl.stanford.edu/papers/asplos12_hong.pdf. Apart from the
> language,
> > there is also a Green-Marl compiler that compiles the language to
> different
> > targets. C++/OpenMP was initially supported, after which support for
> > Stanford GPS (a Pregel-clone developed at Stanford) was added for a
> subset
> > of the features in the Green-Marl language. The big advantage of using a
> > language like Green-Marl is that it enables you to write implicitly
> > parallel programs in an intuitive way with the compiler doing most of the
> > heavy work such as generating the messaging, converting pull-based
> > operation (remote reading) to push-based operations (message sending),
> etc.
> > In the last few months I have worked on a few features in Giraph that
> > enable Green-Marl to target Giraph as a back-end (namely GIRAPH-127,
> > GIRAPH-192 and GIRAPH-216). Since these have all been merged, Green-Marl
> > and Giraph should be compatible as of last week!
> >
> > Using Green-Marl with Giraph is very simple, as long as you have a
> working
> > Hadoop/Giraph environment. Since the Green-Marl compiler is a
> > source-to-source compiler, it is very easy to use it in an existing
> set-up.
> > The steps to using Green-Marl are like this:
> >
> > 1. Download the Green-Marl compiler from
> > https://github.com/stanford-ppl/Green-Marl, run make_dirs.sh in the
> > top-level dir and run make in the src dir. A more detailed
> explanation is
> > given in the Github readme.
> > 2. Compile one of the examples (in the top-level directory) like
> > this: bin/gm_comp -t=giraph apps/src/pagerank.gm
> > 3. Move pagerank.java to the Giraph src directory or your own project,
> > package it into a .jar and go!
> > 4. Explore more examples, modify them and start writing your own
> >
> > Green-Marl is actively being worked on within Oracle Labs, new features
> are
> > being added to the language (such as collection node properties) and
> > support for these features will be added to the distributed back-ends in
> > the near future.
> >
> > In case of questions or suggestions, please send us a message! Feedback
> is
> > very much appreciated and people are more than welcome to contribute
> > everything from small bugfixes to entire back-ends ;-)
> >
> > - Jan
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: Announcing: Green-Marl/Giraph
Posted by Claudio Martella <cl...@gmail.com>.
This is really really cool stuff. congratulations!
On Wed, Jul 11, 2012 at 3:25 AM, Jan van der Lugt
<ja...@oracle.com> wrote:
> Hi everyone,
>
> Some of you might already know Green-Marl, Green-Marl compatibility was
> even mentioned as a new feature for Giraph 0.2, but for those who don't,
> let me give you a quick introduction.
>
> Green-Marl is a domain-specific language tailored to graph algorithms. It
> has many features that makes it possible to write graph algorithms very
> concise and intuitive. For example, it has built-in constructs for graphs,
> node properties, and traversals. A more complete description is given here:
> http://ppl.stanford.edu/papers/asplos12_hong.pdf. Apart from the language,
> there is also a Green-Marl compiler that compiles the language to different
> targets. C++/OpenMP was initially supported, after which support for
> Stanford GPS (a Pregel-clone developed at Stanford) was added for a subset
> of the features in the Green-Marl language. The big advantage of using a
> language like Green-Marl is that it enables you to write implicitly
> parallel programs in an intuitive way with the compiler doing most of the
> heavy work such as generating the messaging, converting pull-based
> operation (remote reading) to push-based operations (message sending), etc.
> In the last few months I have worked on a few features in Giraph that
> enable Green-Marl to target Giraph as a back-end (namely GIRAPH-127,
> GIRAPH-192 and GIRAPH-216). Since these have all been merged, Green-Marl
> and Giraph should be compatible as of last week!
>
> Using Green-Marl with Giraph is very simple, as long as you have a working
> Hadoop/Giraph environment. Since the Green-Marl compiler is a
> source-to-source compiler, it is very easy to use it in an existing set-up.
> The steps to using Green-Marl are like this:
>
> 1. Download the Green-Marl compiler from
> https://github.com/stanford-ppl/Green-Marl, run make_dirs.sh in the
> top-level dir and run make in the src dir. A more detailed explanation is
> given in the Github readme.
> 2. Compile one of the examples (in the top-level directory) like
> this: bin/gm_comp -t=giraph apps/src/pagerank.gm
> 3. Move pagerank.java to the Giraph src directory or your own project,
> package it into a .jar and go!
> 4. Explore more examples, modify them and start writing your own
>
> Green-Marl is actively being worked on within Oracle Labs, new features are
> being added to the language (such as collection node properties) and
> support for these features will be added to the distributed back-ends in
> the near future.
>
> In case of questions or suggestions, please send us a message! Feedback is
> very much appreciated and people are more than welcome to contribute
> everything from small bugfixes to entire back-ends ;-)
>
> - Jan
--
Claudio Martella
claudio.martella@gmail.com
Re: Announcing: Green-Marl/Giraph
Posted by Jan van der Lugt <ja...@gmail.com>.
Hello Praveen,
This very much depends on the algorithm. Due to some recent optimizations
(intra-loop merging) many algorithms should be as fast as hand-implemented
code. Publications on this subject are in preparation. The only Giraph
feature we do not currently use is combiners, but we are thinking about
generating those automatically as well.
- Jan
On Tue, Jul 10, 2012 at 9:14 PM, Praveen Sripati
<pr...@gmail.com>wrote:
> Nice to know about `Green-Marl`. Abstraction would make working with Graphs
> more easy, but also would also add a bit of a performance overhead. Has any
> study been done on the same?
>
> Thanks,
> Praveen
>
>
> On Wed, Jul 11, 2012 at 7:40 AM, Avery Ching <ac...@apache.org> wrote:
>
> > This is really cool stuff.
> >
> > Avery
> >
> >
> > On 7/10/12 6:25 PM, Jan van der Lugt wrote:
> >
> >> Hi everyone,
> >>
> >> Some of you might already know Green-Marl, Green-Marl compatibility was
> >> even mentioned as a new feature for Giraph 0.2, but for those who don't,
> >> let me give you a quick introduction.
> >>
> >> Green-Marl is a domain-specific language tailored to graph algorithms.
> It
> >> has many features that makes it possible to write graph algorithms very
> >> concise and intuitive. For example, it has built-in constructs for
> graphs,
> >> node properties, and traversals. A more complete description is given
> >> here:
> >> http://ppl.stanford.edu/**papers/asplos12_hong.pdf<
> http://ppl.stanford.edu/papers/asplos12_hong.pdf>.
> >> Apart from the language,
> >> there is also a Green-Marl compiler that compiles the language to
> >> different
> >> targets. C++/OpenMP was initially supported, after which support for
> >> Stanford GPS (a Pregel-clone developed at Stanford) was added for a
> subset
> >> of the features in the Green-Marl language. The big advantage of using a
> >> language like Green-Marl is that it enables you to write implicitly
> >> parallel programs in an intuitive way with the compiler doing most of
> the
> >> heavy work such as generating the messaging, converting pull-based
> >> operation (remote reading) to push-based operations (message sending),
> >> etc.
> >> In the last few months I have worked on a few features in Giraph that
> >> enable Green-Marl to target Giraph as a back-end (namely GIRAPH-127,
> >> GIRAPH-192 and GIRAPH-216). Since these have all been merged, Green-Marl
> >> and Giraph should be compatible as of last week!
> >>
> >> Using Green-Marl with Giraph is very simple, as long as you have a
> working
> >> Hadoop/Giraph environment. Since the Green-Marl compiler is a
> >> source-to-source compiler, it is very easy to use it in an existing
> >> set-up.
> >> The steps to using Green-Marl are like this:
> >>
> >> 1. Download the Green-Marl compiler from
> >> https://github.com/stanford-**ppl/Green-Marl<
> https://github.com/stanford-ppl/Green-Marl>,
> >> run make_dirs.sh in the
> >> top-level dir and run make in the src dir. A more detailed
> >> explanation is
> >> given in the Github readme.
> >> 2. Compile one of the examples (in the top-level directory) like
> >> this: bin/gm_comp -t=giraph apps/src/pagerank.gm
> >> 3. Move pagerank.java to the Giraph src directory or your own
> project,
> >> package it into a .jar and go!
> >> 4. Explore more examples, modify them and start writing your own
> >>
> >> Green-Marl is actively being worked on within Oracle Labs, new features
> >> are
> >> being added to the language (such as collection node properties) and
> >> support for these features will be added to the distributed back-ends in
> >> the near future.
> >>
> >> In case of questions or suggestions, please send us a message! Feedback
> is
> >> very much appreciated and people are more than welcome to contribute
> >> everything from small bugfixes to entire back-ends ;-)
> >>
> >> - Jan
> >>
> >>
> >
> >
>
Re: Announcing: Green-Marl/Giraph
Posted by Praveen Sripati <pr...@gmail.com>.
Nice to know about `Green-Marl`. Abstraction would make working with Graphs
more easy, but also would also add a bit of a performance overhead. Has any
study been done on the same?
Thanks,
Praveen
On Wed, Jul 11, 2012 at 7:40 AM, Avery Ching <ac...@apache.org> wrote:
> This is really cool stuff.
>
> Avery
>
>
> On 7/10/12 6:25 PM, Jan van der Lugt wrote:
>
>> Hi everyone,
>>
>> Some of you might already know Green-Marl, Green-Marl compatibility was
>> even mentioned as a new feature for Giraph 0.2, but for those who don't,
>> let me give you a quick introduction.
>>
>> Green-Marl is a domain-specific language tailored to graph algorithms. It
>> has many features that makes it possible to write graph algorithms very
>> concise and intuitive. For example, it has built-in constructs for graphs,
>> node properties, and traversals. A more complete description is given
>> here:
>> http://ppl.stanford.edu/**papers/asplos12_hong.pdf<http://ppl.stanford.edu/papers/asplos12_hong.pdf>.
>> Apart from the language,
>> there is also a Green-Marl compiler that compiles the language to
>> different
>> targets. C++/OpenMP was initially supported, after which support for
>> Stanford GPS (a Pregel-clone developed at Stanford) was added for a subset
>> of the features in the Green-Marl language. The big advantage of using a
>> language like Green-Marl is that it enables you to write implicitly
>> parallel programs in an intuitive way with the compiler doing most of the
>> heavy work such as generating the messaging, converting pull-based
>> operation (remote reading) to push-based operations (message sending),
>> etc.
>> In the last few months I have worked on a few features in Giraph that
>> enable Green-Marl to target Giraph as a back-end (namely GIRAPH-127,
>> GIRAPH-192 and GIRAPH-216). Since these have all been merged, Green-Marl
>> and Giraph should be compatible as of last week!
>>
>> Using Green-Marl with Giraph is very simple, as long as you have a working
>> Hadoop/Giraph environment. Since the Green-Marl compiler is a
>> source-to-source compiler, it is very easy to use it in an existing
>> set-up.
>> The steps to using Green-Marl are like this:
>>
>> 1. Download the Green-Marl compiler from
>> https://github.com/stanford-**ppl/Green-Marl<https://github.com/stanford-ppl/Green-Marl>,
>> run make_dirs.sh in the
>> top-level dir and run make in the src dir. A more detailed
>> explanation is
>> given in the Github readme.
>> 2. Compile one of the examples (in the top-level directory) like
>> this: bin/gm_comp -t=giraph apps/src/pagerank.gm
>> 3. Move pagerank.java to the Giraph src directory or your own project,
>> package it into a .jar and go!
>> 4. Explore more examples, modify them and start writing your own
>>
>> Green-Marl is actively being worked on within Oracle Labs, new features
>> are
>> being added to the language (such as collection node properties) and
>> support for these features will be added to the distributed back-ends in
>> the near future.
>>
>> In case of questions or suggestions, please send us a message! Feedback is
>> very much appreciated and people are more than welcome to contribute
>> everything from small bugfixes to entire back-ends ;-)
>>
>> - Jan
>>
>>
>
>
Re: Announcing: Green-Marl/Giraph
Posted by Avery Ching <ac...@apache.org>.
This is really cool stuff.
Avery
On 7/10/12 6:25 PM, Jan van der Lugt wrote:
> Hi everyone,
>
> Some of you might already know Green-Marl, Green-Marl compatibility was
> even mentioned as a new feature for Giraph 0.2, but for those who don't,
> let me give you a quick introduction.
>
> Green-Marl is a domain-specific language tailored to graph algorithms. It
> has many features that makes it possible to write graph algorithms very
> concise and intuitive. For example, it has built-in constructs for graphs,
> node properties, and traversals. A more complete description is given here:
> http://ppl.stanford.edu/papers/asplos12_hong.pdf. Apart from the language,
> there is also a Green-Marl compiler that compiles the language to different
> targets. C++/OpenMP was initially supported, after which support for
> Stanford GPS (a Pregel-clone developed at Stanford) was added for a subset
> of the features in the Green-Marl language. The big advantage of using a
> language like Green-Marl is that it enables you to write implicitly
> parallel programs in an intuitive way with the compiler doing most of the
> heavy work such as generating the messaging, converting pull-based
> operation (remote reading) to push-based operations (message sending), etc.
> In the last few months I have worked on a few features in Giraph that
> enable Green-Marl to target Giraph as a back-end (namely GIRAPH-127,
> GIRAPH-192 and GIRAPH-216). Since these have all been merged, Green-Marl
> and Giraph should be compatible as of last week!
>
> Using Green-Marl with Giraph is very simple, as long as you have a working
> Hadoop/Giraph environment. Since the Green-Marl compiler is a
> source-to-source compiler, it is very easy to use it in an existing set-up.
> The steps to using Green-Marl are like this:
>
> 1. Download the Green-Marl compiler from
> https://github.com/stanford-ppl/Green-Marl, run make_dirs.sh in the
> top-level dir and run make in the src dir. A more detailed explanation is
> given in the Github readme.
> 2. Compile one of the examples (in the top-level directory) like
> this: bin/gm_comp -t=giraph apps/src/pagerank.gm
> 3. Move pagerank.java to the Giraph src directory or your own project,
> package it into a .jar and go!
> 4. Explore more examples, modify them and start writing your own
>
> Green-Marl is actively being worked on within Oracle Labs, new features are
> being added to the language (such as collection node properties) and
> support for these features will be added to the distributed back-ends in
> the near future.
>
> In case of questions or suggestions, please send us a message! Feedback is
> very much appreciated and people are more than welcome to contribute
> everything from small bugfixes to entire back-ends ;-)
>
> - Jan
>