You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Josh Patterson <jo...@cloudera.com> on 2011/09/01 18:26:37 UTC
Re: Re:Mahout compatibility with GraphLab
Danny,
(Only tangentially understanding the internals of GraphLab) --- is
there any interest in having this framework run as a first class
citizen on hadoop (beside MR) similar to:
MPI on hadoop
https://issues.apache.org/jira/browse/MAPREDUCE-2911
via the new MRv2 framework? I think that would be a really interesting
way to get GraphLab to work with Mahout as I see MRv2 as enabling a
lot more functionality in Mahout as we move forward.
Thoughts?
JP
On Sun, Aug 28, 2011 at 10:44 AM, Danny Bickson <da...@gmail.com> wrote:
> Yes, it is not using map reduce - we have our own communication
> infrastructure written in C++.
> You can learn more by looking at http://graphlab.org
>
> 2011/8/28 myn <my...@163.com>
>
>> Is that a non hadoop mapreduce project?
>>
>>
>> At 2011-08-28 21:32:51,myn <my...@163.com> wrote:
>>
>>
>>
>> sorry i can`t visit the page
>>
>> http://bickson.blogspot.com/2011/08/graphlab-and-mahout-compatability.html
>>
>> At 2011-08-28 20:13:06,"Danny Bickson" <da...@gmail.com> wrote:
>> >Hi Mahouters!
>> >In the last couple of months we are trying to improve CMU GraphLab's large
>> >scale machine learning (http://graphlab.org) compatibility with Mahout.
>> >As a first step, we changed our open source license to Apache. Now we are
>> >working on writing some of the GraphLab wrapper code in Java,
>> >thus allowing for better interaction between the projects. Our vision is
>> to
>> >use Mahout for the inherently parallel tasks for large scale data it is
>> very
>> >good at,
>> >while having GraphLab functionality of more complex interactions of data.
>> >
>> >I am looking for any Mahout users who have large scale matrices that are
>> >factorized using Mahout SVD, who want to try out GraphLab SVD solver.
>> >I wrote some code to allow for easy transition between Mahout
>> >SequetialAccessSparseVector to GraphLab's binary format. If anyone wants
>> to
>> >try
>> >it out, instructions are found on:
>> >
>> http://bickson.blogspot.com/2011/08/graphlab-and-mahout-compatability.html
>> >Any feedback is welcome!
>> >
>> >- Danny Bickson
>>
>>
>>
>>
>
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
Re: Re:Mahout compatibility with GraphLab
Posted by Ted Dunning <te...@gmail.com>.
The 0.23 release branch has been cut, but last I checked (a week or two
ago), YARN was still at the stage of leaking resources. To me that implies
the need for caution because that usually occurs due to bad lock hygiene.
That sort of problem really is only reparable with careful and correct
design and once cleanliness is lost, it is very difficult to regain.
On the other hand, my opinion of the life stages for software is VAPOR =>
DESIGN => (BROKEN => FIXED)+ => REALLY_FIXED. As such, BROKEN is a
necessary step on the road to REALLY_FIXED. Software is the most visible
expression of original sin, you might say. Faith and good works are
required to get beyond that point.
Regarding C++, I am pretty sure that is in the road map and probably works
at some level already since so many of the use cases for MR2.0 involve
distributing non-java programs.
On Thu, Sep 1, 2011 at 5:18 PM, Joseph Gonzalez <joseph.e.gonzalez@gmail.com
> wrote:
> Does anyone know the status of YARN? Is it ready for launching/managing
> C++ jobs? I have been looking for a project page but have not been very
> successful.
>
Re: Re:Mahout compatibility with GraphLab
Posted by Ted Dunning <te...@gmail.com>.
Well, protobufs is always an option as well.
There was a bit of a tiff between Yahoo developers and Cloudera over Avro a
few months ago. There may be a bit of a residual bias left over from that
history. Much of the motive force behind MR2.0 is now at Hortonworks so you
could imagine some momentum persisted across the event horizon surrounding
the exit from Yahoo.
(caveat, I have good friends who use protobufs and friends who use Avro. I
like them both for different reasons)
On Thu, Sep 1, 2011 at 5:18 PM, Joseph Gonzalez <joseph.e.gonzalez@gmail.com
> wrote:
> With respect to MRv2 I think there are still some issues that need to be
> resolved. For example I looked at the pipeline:
>
> Hadoop Graph Creation --> Hadoop/Yarn GraphLab Launch --> Hadoop Post
> Processing
>
> This requires a common data format. I constructed a prototype around AVRO
> but found that the C++ implementation lacks nested structures which I
> "needed" to cleanly encode the GraphLab data graph. Also, surprisingly,
> MRv2 did not seem to have an AVRO interface (maybe this is fixed now).
>
> While these problems are not insurmountable they make it difficult to
> cleanly integrate with Hadoop, Yarn, and Avro while they are transitioning
> to MRv2 (especially when the C++bindings seem to be updated more slowly).
>
Re: Re:Mahout compatibility with GraphLab
Posted by Joseph Gonzalez <jo...@gmail.com>.
Hi Everyone,
I have been looking at GraphLab integration with Hadoop while visiting
Yahoo! Research over the summer. It seems increasingly clear with the
convergence of cloud tools to the Apache framework that some form of
integration will be extremely helpful. However, determining the right path
has been surprisingly challenging.
As the presentation (great link) notes, Hadoop is being split into cluster
management/scheduling tools (i.e., YARN) and the MapReduce computational
model (i.e., Hadoop). In that setting GraphLab will, ideally, run on-top of
YARN (not Hadoop) and use the exposed distributed file system resources to
load and save graphs. While it is possible to launch GraphLab from within
Hadoop streaming, as a map-only job, this is slightly less clean and appears
to be the old way of integrating external toolkits.
Does anyone know the status of YARN? Is it ready for launching/managing C++
jobs? I have been looking for a project page but have not been very
successful.
With respect to MRv2 I think there are still some issues that need to be
resolved. For example I looked at the pipeline:
Hadoop Graph Creation --> Hadoop/Yarn GraphLab Launch --> Hadoop Post
Processing
This requires a common data format. I constructed a prototype around AVRO
but found that the C++ implementation lacks nested structures which I
"needed" to cleanly encode the GraphLab data graph. Also, surprisingly,
MRv2 did not seem to have an AVRO interface (maybe this is fixed now).
While these problems are not insurmountable they make it difficult to
cleanly integrate with Hadoop, Yarn, and Avro while they are transitioning
to MRv2 (especially when the C++bindings seem to be updated more slowly).
An alternative, potentially cleaner, strategy might be to try to build a
Java (or at least JVM based) version of GraphLab. This would eliminate the
need for cross language formats and a JNI interface for Java based update
functions. However, maintaining two versions of GraphLab could be difficult
and we would probably take a performance hit.
We are now in the processing of building a roadmap for the next generation
of GraphLab which will provide more powerful abstractions, improved
scalability, and greater ease-of-use. Any feedback, suggestions, or ideas
would be great!
Joey
PS: We are co-organizing a NIPS'11 workshop (http://biglearn.org) which will
address issues related to large-scale machine learning and will provide an
excellent venue to discuss future plans and system integration.
Re: Re:Mahout compatibility with GraphLab
Posted by Ted Dunning <te...@gmail.com>.
SOunds like a comparison with some un-named alternative.
What requires C++ to java wrappers?
On Thu, Sep 1, 2011 at 12:39 PM, Josh Patterson <jo...@cloudera.com> wrote:
> Quick presentation on the framework by Sharad Agarwal (Yahoo):
>
> https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributors_meet_07_01_2011.pdf
>
> Seems to be a considerably more native/clean approach than managing a
> bunch of C++ to Java wrappers, imho.
>
Re: Re:Mahout compatibility with GraphLab
Posted by Ahmed Radwan <ah...@cloudera.com>.
I'll definitly encourage this integration. The architecture document at:
https://issues.apache.org/jira/secure/attachment/12486023/MapReduce_NextGen_Architecture.pdf
captures
a lot of details.
Best Regards
Ahmed
On Thu, Sep 1, 2011 at 10:39 AM, Josh Patterson <jo...@cloudera.com> wrote:
> From the Cloudera side, there is also Ahmed Radwan, who I'm sure would
> answer questions you may have.
>
> JP
>
> On Thu, Sep 1, 2011 at 1:39 PM, Josh Patterson <jo...@cloudera.com> wrote:
> > This feature seems to be principly driven by a few folks @ Yahoo. Arun
> > Murthy seems to be quite involved.
> >
> > Hama is also getting into the MRv2 game:
> > https://issues.apache.org/jira/browse/HAMA-431
> >
> > Quick presentation on the framework by Sharad Agarwal (Yahoo):
> >
> https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributors_meet_07_01_2011.pdf
> >
> > Seems to be a considerably more native/clean approach than managing a
> > bunch of C++ to Java wrappers, imho. I'd really like to see a lot of
> > frameworks on Hadoop; could have a language like Pig that then
> > compiled down not only to different ML techniques, but also execute
> > them on the proper framework in MRv2 as well.
> >
> > JP
> >
> > On Thu, Sep 1, 2011 at 1:12 PM, Danny Bickson <da...@gmail.com>
> wrote:
> >> This is definitely a very interesting option to integrate both projects.
> I
> >> will forward this to my
> >> collaborators - specifically, Joey Gonzalez was recently an Intern at
> Yahoo!
> >> and looked at similar directions.
> >> I will try to learn the technical details better to see how this
> >> interoperability works.
> >> Who is working on this project in case we need more details?
> >>
> >> Thanks,
> >>
> >> Danny Bickson
> >>
> >> On Thu, Sep 1, 2011 at 7:26 PM, Josh Patterson <jo...@cloudera.com>
> wrote:
> >>
> >>> Danny,
> >>> (Only tangentially understanding the internals of GraphLab) --- is
> >>> there any interest in having this framework run as a first class
> >>> citizen on hadoop (beside MR) similar to:
> >>>
> >>> MPI on hadoop
> >>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
> >>>
> >>> via the new MRv2 framework? I think that would be a really interesting
> >>> way to get GraphLab to work with Mahout as I see MRv2 as enabling a
> >>> lot more functionality in Mahout as we move forward.
> >>>
> >>> Thoughts?
> >>>
> >>> JP
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > Twitter: @jpatanooga
> > Solution Architect @ Cloudera
> > hadoop: http://www.cloudera.com
> >
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
>
Re: Re:Mahout compatibility with GraphLab
Posted by Josh Patterson <jo...@cloudera.com>.
>From the Cloudera side, there is also Ahmed Radwan, who I'm sure would
answer questions you may have.
JP
On Thu, Sep 1, 2011 at 1:39 PM, Josh Patterson <jo...@cloudera.com> wrote:
> This feature seems to be principly driven by a few folks @ Yahoo. Arun
> Murthy seems to be quite involved.
>
> Hama is also getting into the MRv2 game:
> https://issues.apache.org/jira/browse/HAMA-431
>
> Quick presentation on the framework by Sharad Agarwal (Yahoo):
> https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributors_meet_07_01_2011.pdf
>
> Seems to be a considerably more native/clean approach than managing a
> bunch of C++ to Java wrappers, imho. I'd really like to see a lot of
> frameworks on Hadoop; could have a language like Pig that then
> compiled down not only to different ML techniques, but also execute
> them on the proper framework in MRv2 as well.
>
> JP
>
> On Thu, Sep 1, 2011 at 1:12 PM, Danny Bickson <da...@gmail.com> wrote:
>> This is definitely a very interesting option to integrate both projects. I
>> will forward this to my
>> collaborators - specifically, Joey Gonzalez was recently an Intern at Yahoo!
>> and looked at similar directions.
>> I will try to learn the technical details better to see how this
>> interoperability works.
>> Who is working on this project in case we need more details?
>>
>> Thanks,
>>
>> Danny Bickson
>>
>> On Thu, Sep 1, 2011 at 7:26 PM, Josh Patterson <jo...@cloudera.com> wrote:
>>
>>> Danny,
>>> (Only tangentially understanding the internals of GraphLab) --- is
>>> there any interest in having this framework run as a first class
>>> citizen on hadoop (beside MR) similar to:
>>>
>>> MPI on hadoop
>>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>>
>>> via the new MRv2 framework? I think that would be a really interesting
>>> way to get GraphLab to work with Mahout as I see MRv2 as enabling a
>>> lot more functionality in Mahout as we move forward.
>>>
>>> Thoughts?
>>>
>>> JP
>>>
>>>
>>
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
>
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
Re: Re:Mahout compatibility with GraphLab
Posted by Ted Dunning <te...@gmail.com>.
Forgive my skepticism, but I would rather not base anything on Hama.
On Thu, Sep 1, 2011 at 12:39 PM, Josh Patterson <jo...@cloudera.com> wrote:
> Hama is also getting into the MRv2 game:
> https://issues.apache.org/jira/browse/HAMA-431
>
Re: Re:Mahout compatibility with GraphLab
Posted by Josh Patterson <jo...@cloudera.com>.
This feature seems to be principly driven by a few folks @ Yahoo. Arun
Murthy seems to be quite involved.
Hama is also getting into the MRv2 game:
https://issues.apache.org/jira/browse/HAMA-431
Quick presentation on the framework by Sharad Agarwal (Yahoo):
https://issues.apache.org/jira/secure/attachment/12485267/hadoop_contributors_meet_07_01_2011.pdf
Seems to be a considerably more native/clean approach than managing a
bunch of C++ to Java wrappers, imho. I'd really like to see a lot of
frameworks on Hadoop; could have a language like Pig that then
compiled down not only to different ML techniques, but also execute
them on the proper framework in MRv2 as well.
JP
On Thu, Sep 1, 2011 at 1:12 PM, Danny Bickson <da...@gmail.com> wrote:
> This is definitely a very interesting option to integrate both projects. I
> will forward this to my
> collaborators - specifically, Joey Gonzalez was recently an Intern at Yahoo!
> and looked at similar directions.
> I will try to learn the technical details better to see how this
> interoperability works.
> Who is working on this project in case we need more details?
>
> Thanks,
>
> Danny Bickson
>
> On Thu, Sep 1, 2011 at 7:26 PM, Josh Patterson <jo...@cloudera.com> wrote:
>
>> Danny,
>> (Only tangentially understanding the internals of GraphLab) --- is
>> there any interest in having this framework run as a first class
>> citizen on hadoop (beside MR) similar to:
>>
>> MPI on hadoop
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>> via the new MRv2 framework? I think that would be a really interesting
>> way to get GraphLab to work with Mahout as I see MRv2 as enabling a
>> lot more functionality in Mahout as we move forward.
>>
>> Thoughts?
>>
>> JP
>>
>>
>
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
Re: Re:Mahout compatibility with GraphLab
Posted by Danny Bickson <da...@gmail.com>.
This is definitely a very interesting option to integrate both projects. I
will forward this to my
collaborators - specifically, Joey Gonzalez was recently an Intern at Yahoo!
and looked at similar directions.
I will try to learn the technical details better to see how this
interoperability works.
Who is working on this project in case we need more details?
Thanks,
Danny Bickson
On Thu, Sep 1, 2011 at 7:26 PM, Josh Patterson <jo...@cloudera.com> wrote:
> Danny,
> (Only tangentially understanding the internals of GraphLab) --- is
> there any interest in having this framework run as a first class
> citizen on hadoop (beside MR) similar to:
>
> MPI on hadoop
> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>
> via the new MRv2 framework? I think that would be a really interesting
> way to get GraphLab to work with Mahout as I see MRv2 as enabling a
> lot more functionality in Mahout as we move forward.
>
> Thoughts?
>
> JP
>
>
Re: Re:Mahout compatibility with GraphLab
Posted by Ted Dunning <te...@gmail.com>.
MRv2 could be huge for Mahout.
+1 for enabling additional computational models as they become feasible.
On Thu, Sep 1, 2011 at 11:26 AM, Josh Patterson <jo...@cloudera.com> wrote:
> Danny,
> (Only tangentially understanding the internals of GraphLab) --- is
> there any interest in having this framework run as a first class
> citizen on hadoop (beside MR) similar to:
>
> MPI on hadoop
> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>
> via the new MRv2 framework? I think that would be a really interesting
> way to get GraphLab to work with Mahout as I see MRv2 as enabling a
> lot more functionality in Mahout as we move forward.
>
> Thoughts?
>
> JP
>
> On Sun, Aug 28, 2011 at 10:44 AM, Danny Bickson <da...@gmail.com>
> wrote:
> > Yes, it is not using map reduce - we have our own communication
> > infrastructure written in C++.
> > You can learn more by looking at http://graphlab.org
> >
> > 2011/8/28 myn <my...@163.com>
> >
> >> Is that a non hadoop mapreduce project?
> >>
> >>
> >> At 2011-08-28 21:32:51,myn <my...@163.com> wrote:
> >>
> >>
> >>
> >> sorry i can`t visit the page
> >>
> >>
> http://bickson.blogspot.com/2011/08/graphlab-and-mahout-compatability.html
> >>
> >> At 2011-08-28 20:13:06,"Danny Bickson" <da...@gmail.com> wrote:
> >> >Hi Mahouters!
> >> >In the last couple of months we are trying to improve CMU GraphLab's
> large
> >> >scale machine learning (http://graphlab.org) compatibility with
> Mahout.
> >> >As a first step, we changed our open source license to Apache. Now we
> are
> >> >working on writing some of the GraphLab wrapper code in Java,
> >> >thus allowing for better interaction between the projects. Our vision
> is
> >> to
> >> >use Mahout for the inherently parallel tasks for large scale data it is
> >> very
> >> >good at,
> >> >while having GraphLab functionality of more complex interactions of
> data.
> >> >
> >> >I am looking for any Mahout users who have large scale matrices that
> are
> >> >factorized using Mahout SVD, who want to try out GraphLab SVD solver.
> >> >I wrote some code to allow for easy transition between Mahout
> >> >SequetialAccessSparseVector to GraphLab's binary format. If anyone
> wants
> >> to
> >> >try
> >> >it out, instructions are found on:
> >> >
> >>
> http://bickson.blogspot.com/2011/08/graphlab-and-mahout-compatability.html
> >> >Any feedback is welcome!
> >> >
> >> >- Danny Bickson
> >>
> >>
> >>
> >>
> >
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
>