You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ryan Compton <co...@gmail.com> on 2013/08/08 05:54:40 UTC
Re: Bagel?
I'm interested in processing graphs with Spark. I'm able to do sbt/sbt
assembly and the process exits with success. I can find:
-bash-3.2$ find . | grep jar$ | grep -assembly-
./core/target/spark-core-assembly-0.7.3cdh3u3.jar
./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
I add these to my local Nexus repo, include them in my pom.xml, and
build/run distributed jobs just fine.
Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
didn't make a Bagel assembly.
Also, I can't find example code. The code here:
https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
of @serializable depreciation warnings and I can't find the ./run
script they are taking about at the bottom of this page:
http://spark-project.org/docs/latest/bagel-programming-guide.html
I get the impression that Bagel may be discontinued. I also noticed
some ads for something called "GraphX" on the BDAS website. Should I
be using GraphX instead?
On Wed, Aug 7, 2013 at 7:05 PM, Ryan Compton <co...@gmail.com> wrote:
> I'm interested in processing graphs with Spark. I'm able to do
> sbt/sbt assembly and the process exits with success. I can find:
>
> -bash-3.2$ find . | grep jar$ | grep -assembly-
> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>
> I add these to my local Nexus repo, include them in my pom.xml, and
> build/run distributed jobs just fine.
>
> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
> didn't make a Bagel assembly.
>
> Also, I can't find example code. The code here:
> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
> of @serializable depreciation warnings and I can't find the ./run
> script they are taking about at the bottom of this page:
> http://spark-project.org/docs/latest/bagel-programming-guide.html
>
> I get the impression that Bagel may be discontinued. Is there
> something else I can use for graphs?
>
> On Wed, Aug 7, 2013 at 7:02 PM, Ryan Compton <co...@gmail.com> wrote:
>> I'm interesting in processing graphs with Spark. I'm able to do
>> sbt/sbt assembly and the process exits with success. I can find:
>>
>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>
>> I add these to my local Nexus repo, include them in my pom.xml, and
>> build/run distributed jobs just fine.
>>
>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>> didn't make a Bagel assembly.
>>
>> Also, I can't find example code. The code here:
>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>> of @serializable depreciation warnings and I can't find the ./run
>> script they are taking about at the bottom of this page:
>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>
>> I get the impression that Bagel may be discontinued. Is there
>> something else I can use for graphs?
Re: Bagel?
Posted by Matei Zaharia <ma...@gmail.com>.
Hmm, it's weird that it built two. It should just be spark-0.7.3/bagel/target.
Matei
On Aug 14, 2013, at 2:29 PM, Ryan Compton <co...@gmail.com> wrote:
> spark-0.7.3/bagel/target or spark-0.7.3/bagel/bagel/target ?
>
> On Wed, Aug 7, 2013 at 9:17 PM, Matei Zaharia <ma...@gmail.com> wrote:
>> Hi Ryan,
>>
>> To use Bagel, you can just add the spark-bagel.jar built within the bagel/target directory, in addition to the spark-core-assembly. It doesn't have other dependencies beyond Spark Core.
>>
>> GraphX is an upcoming project to do richer types of graph processing, but it's fine to use Bagel for now. We may eventually update the Bagel API to run on top of GraphX.
>>
>> Matei
>>
>> On Aug 7, 2013, at 8:54 PM, Ryan Compton <co...@gmail.com> wrote:
>>
>>> I'm interested in processing graphs with Spark. I'm able to do sbt/sbt
>>> assembly and the process exits with success. I can find:
>>>
>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>
>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>> build/run distributed jobs just fine.
>>>
>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>> didn't make a Bagel assembly.
>>>
>>> Also, I can't find example code. The code here:
>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>> of @serializable depreciation warnings and I can't find the ./run
>>> script they are taking about at the bottom of this page:
>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>
>>> I get the impression that Bagel may be discontinued. I also noticed
>>> some ads for something called "GraphX" on the BDAS website. Should I
>>> be using GraphX instead?
>>>
>>> On Wed, Aug 7, 2013 at 7:05 PM, Ryan Compton <co...@gmail.com> wrote:
>>>> I'm interested in processing graphs with Spark. I'm able to do
>>>> sbt/sbt assembly and the process exits with success. I can find:
>>>>
>>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>>
>>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>>> build/run distributed jobs just fine.
>>>>
>>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>>> didn't make a Bagel assembly.
>>>>
>>>> Also, I can't find example code. The code here:
>>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>>> of @serializable depreciation warnings and I can't find the ./run
>>>> script they are taking about at the bottom of this page:
>>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>>
>>>> I get the impression that Bagel may be discontinued. Is there
>>>> something else I can use for graphs?
>>>>
>>>> On Wed, Aug 7, 2013 at 7:02 PM, Ryan Compton <co...@gmail.com> wrote:
>>>>> I'm interesting in processing graphs with Spark. I'm able to do
>>>>> sbt/sbt assembly and the process exits with success. I can find:
>>>>>
>>>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>>>
>>>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>>>> build/run distributed jobs just fine.
>>>>>
>>>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>>>> didn't make a Bagel assembly.
>>>>>
>>>>> Also, I can't find example code. The code here:
>>>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>>>> of @serializable depreciation warnings and I can't find the ./run
>>>>> script they are taking about at the bottom of this page:
>>>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>>>
>>>>> I get the impression that Bagel may be discontinued. Is there
>>>>> something else I can use for graphs?
>>
Re: Bagel?
Posted by Ryan Compton <co...@gmail.com>.
spark-0.7.3/bagel/target or spark-0.7.3/bagel/bagel/target ?
On Wed, Aug 7, 2013 at 9:17 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Hi Ryan,
>
> To use Bagel, you can just add the spark-bagel.jar built within the bagel/target directory, in addition to the spark-core-assembly. It doesn't have other dependencies beyond Spark Core.
>
> GraphX is an upcoming project to do richer types of graph processing, but it's fine to use Bagel for now. We may eventually update the Bagel API to run on top of GraphX.
>
> Matei
>
> On Aug 7, 2013, at 8:54 PM, Ryan Compton <co...@gmail.com> wrote:
>
>> I'm interested in processing graphs with Spark. I'm able to do sbt/sbt
>> assembly and the process exits with success. I can find:
>>
>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>
>> I add these to my local Nexus repo, include them in my pom.xml, and
>> build/run distributed jobs just fine.
>>
>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>> didn't make a Bagel assembly.
>>
>> Also, I can't find example code. The code here:
>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>> of @serializable depreciation warnings and I can't find the ./run
>> script they are taking about at the bottom of this page:
>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>
>> I get the impression that Bagel may be discontinued. I also noticed
>> some ads for something called "GraphX" on the BDAS website. Should I
>> be using GraphX instead?
>>
>> On Wed, Aug 7, 2013 at 7:05 PM, Ryan Compton <co...@gmail.com> wrote:
>>> I'm interested in processing graphs with Spark. I'm able to do
>>> sbt/sbt assembly and the process exits with success. I can find:
>>>
>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>
>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>> build/run distributed jobs just fine.
>>>
>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>> didn't make a Bagel assembly.
>>>
>>> Also, I can't find example code. The code here:
>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>> of @serializable depreciation warnings and I can't find the ./run
>>> script they are taking about at the bottom of this page:
>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>
>>> I get the impression that Bagel may be discontinued. Is there
>>> something else I can use for graphs?
>>>
>>> On Wed, Aug 7, 2013 at 7:02 PM, Ryan Compton <co...@gmail.com> wrote:
>>>> I'm interesting in processing graphs with Spark. I'm able to do
>>>> sbt/sbt assembly and the process exits with success. I can find:
>>>>
>>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>>
>>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>>> build/run distributed jobs just fine.
>>>>
>>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>>> didn't make a Bagel assembly.
>>>>
>>>> Also, I can't find example code. The code here:
>>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>>> of @serializable depreciation warnings and I can't find the ./run
>>>> script they are taking about at the bottom of this page:
>>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>>
>>>> I get the impression that Bagel may be discontinued. Is there
>>>> something else I can use for graphs?
>
Re: Bagel?
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Ryan,
To use Bagel, you can just add the spark-bagel.jar built within the bagel/target directory, in addition to the spark-core-assembly. It doesn't have other dependencies beyond Spark Core.
GraphX is an upcoming project to do richer types of graph processing, but it's fine to use Bagel for now. We may eventually update the Bagel API to run on top of GraphX.
Matei
On Aug 7, 2013, at 8:54 PM, Ryan Compton <co...@gmail.com> wrote:
> I'm interested in processing graphs with Spark. I'm able to do sbt/sbt
> assembly and the process exits with success. I can find:
>
> -bash-3.2$ find . | grep jar$ | grep -assembly-
> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>
> I add these to my local Nexus repo, include them in my pom.xml, and
> build/run distributed jobs just fine.
>
> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
> didn't make a Bagel assembly.
>
> Also, I can't find example code. The code here:
> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
> of @serializable depreciation warnings and I can't find the ./run
> script they are taking about at the bottom of this page:
> http://spark-project.org/docs/latest/bagel-programming-guide.html
>
> I get the impression that Bagel may be discontinued. I also noticed
> some ads for something called "GraphX" on the BDAS website. Should I
> be using GraphX instead?
>
> On Wed, Aug 7, 2013 at 7:05 PM, Ryan Compton <co...@gmail.com> wrote:
>> I'm interested in processing graphs with Spark. I'm able to do
>> sbt/sbt assembly and the process exits with success. I can find:
>>
>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>
>> I add these to my local Nexus repo, include them in my pom.xml, and
>> build/run distributed jobs just fine.
>>
>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>> didn't make a Bagel assembly.
>>
>> Also, I can't find example code. The code here:
>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>> of @serializable depreciation warnings and I can't find the ./run
>> script they are taking about at the bottom of this page:
>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>
>> I get the impression that Bagel may be discontinued. Is there
>> something else I can use for graphs?
>>
>> On Wed, Aug 7, 2013 at 7:02 PM, Ryan Compton <co...@gmail.com> wrote:
>>> I'm interesting in processing graphs with Spark. I'm able to do
>>> sbt/sbt assembly and the process exits with success. I can find:
>>>
>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>
>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>> build/run distributed jobs just fine.
>>>
>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>> didn't make a Bagel assembly.
>>>
>>> Also, I can't find example code. The code here:
>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>> of @serializable depreciation warnings and I can't find the ./run
>>> script they are taking about at the bottom of this page:
>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>
>>> I get the impression that Bagel may be discontinued. Is there
>>> something else I can use for graphs?