You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adaryl Wakefield <ad...@hotmail.com> on 2017/07/24 23:08:45 UTC
real world spark code
Anybody know of publicly available GitHub repos of real world Spark applications written in scala?
Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.massstreet.net<http://www.massstreet.net/>
www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba>
Twitter: @BobLovesData<http://twitter.com/BobLovesData>
Re: real world spark code
Posted by Matei Zaharia <ma...@gmail.com>.
You can also find a lot of GitHub repos for external packages here: http://spark.apache.org/third-party-projects.html
Matei
> On Jul 25, 2017, at 5:30 PM, Frank Austin Nothaft <fn...@berkeley.edu> wrote:
>
> There’s a number of real-world open source Spark applications in the sciences:
>
> genomics:
>
> github.com/bigdatagenomics/adam <— core is scala, has py/r wrappers
> https://github.com/broadinstitute/gatk <— core is java
> https://github.com/hail-is/hail <— core is scala, mostly used through python wrappers
>
> neuroscience:
>
> https://github.com/thunder-project/thunder#using-with-spark <— pyspark
>
> Frank Austin Nothaft
> fnothaft@berkeley.edu
> fnothaft@eecs.berkeley.edu
> 202-340-0466
>
>> On Jul 25, 2017, at 8:09 AM, Jörn Franke <jo...@gmail.com> wrote:
>>
>> Continuous integration (Travis, jenkins) and reporting on unit tests, integration tests etc for each source code version.
>>
>> On 25. Jul 2017, at 16:58, Adaryl Wakefield <ad...@hotmail.com> wrote:
>>
>>> ci+reporting? I’ve never heard of that term before. What is that?
>>>
>>> Adaryl "Bob" Wakefield, MBA
>>> Principal
>>> Mass Street Analytics, LLC
>>> 913.938.6685
>>> www.massstreet.net
>>> www.linkedin.com/in/bobwakefieldmba
>>> Twitter: @BobLovesData
>>>
>>>
>>> From: Jörn Franke [mailto:jornfranke@gmail.com]
>>> Sent: Tuesday, July 25, 2017 8:31 AM
>>> To: Adaryl Wakefield <ad...@hotmail.com>
>>> Cc: user@spark.apache.org
>>> Subject: Re: real world spark code
>>>
>>> Look for the ones that have unit and integration tests as well as a ci+reporting on code quality.
>>>
>>> All the others are just toy examples. Well should be :)
>>>
>>> On 25. Jul 2017, at 01:08, Adaryl Wakefield <ad...@hotmail.com> wrote:
>>>
>>> Anybody know of publicly available GitHub repos of real world Spark applications written in scala?
>>>
>>> Adaryl "Bob" Wakefield, MBA
>>> Principal
>>> Mass Street Analytics, LLC
>>> 913.938.6685
>>> www.massstreet.net
>>> www.linkedin.com/in/bobwakefieldmba
>>> Twitter: @BobLovesData
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: real world spark code
Posted by Frank Austin Nothaft <fn...@berkeley.edu>.
There’s a number of real-world open source Spark applications in the sciences:
genomics:
github.com/bigdatagenomics/adam <http://github.com/bigdatagenomics/adam> <— core is scala, has py/r wrappers
https://github.com/broadinstitute/gatk <https://github.com/broadinstitute/gatk> <— core is java
https://github.com/hail-is/hail <https://github.com/hail-is/hail> <— core is scala, mostly used through python wrappers
neuroscience:
https://github.com/thunder-project/thunder#using-with-spark <https://github.com/thunder-project/thunder#using-with-spark> <— pyspark
Frank Austin Nothaft
fnothaft@berkeley.edu
fnothaft@eecs.berkeley.edu
202-340-0466
> On Jul 25, 2017, at 8:09 AM, Jörn Franke <jo...@gmail.com> wrote:
>
> Continuous integration (Travis, jenkins) and reporting on unit tests, integration tests etc for each source code version.
>
> On 25. Jul 2017, at 16:58, Adaryl Wakefield <adaryl.wakefield@hotmail.com <ma...@hotmail.com>> wrote:
>
>> ci+reporting? I’ve never heard of that term before. What is that?
>>
>> Adaryl "Bob" Wakefield, MBA
>> Principal
>> Mass Street Analytics, LLC
>> 913.938.6685
>> www.massstreet.net <http://www.massstreet.net/>
>> www.linkedin.com/in/bobwakefieldmba <http://www.linkedin.com/in/bobwakefieldmba>
>> Twitter: @BobLovesData <http://twitter.com/BobLovesData>
>>
>>
>> From: Jörn Franke [mailto:jornfranke@gmail.com <ma...@gmail.com>]
>> Sent: Tuesday, July 25, 2017 8:31 AM
>> To: Adaryl Wakefield <adaryl.wakefield@hotmail.com <ma...@hotmail.com>>
>> Cc: user@spark.apache.org <ma...@spark.apache.org>
>> Subject: Re: real world spark code
>>
>> Look for the ones that have unit and integration tests as well as a ci+reporting on code quality.
>>
>> All the others are just toy examples. Well should be :)
>>
>> On 25. Jul 2017, at 01:08, Adaryl Wakefield <adaryl.wakefield@hotmail.com <ma...@hotmail.com>> wrote:
>>
>> Anybody know of publicly available GitHub repos of real world Spark applications written in scala?
>>
>> Adaryl "Bob" Wakefield, MBA
>> Principal
>> Mass Street Analytics, LLC
>> 913.938.6685
>> www.massstreet.net <http://www.massstreet.net/>
>> www.linkedin.com/in/bobwakefieldmba <http://www.linkedin.com/in/bobwakefieldmba>
>> Twitter: @BobLovesData <http://twitter.com/BobLovesData>
Re: real world spark code
Posted by Jörn Franke <jo...@gmail.com>.
Continuous integration (Travis, jenkins) and reporting on unit tests, integration tests etc for each source code version.
> On 25. Jul 2017, at 16:58, Adaryl Wakefield <ad...@hotmail.com> wrote:
>
> ci+reporting? I’ve never heard of that term before. What is that?
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.massstreet.net
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
>
> From: Jörn Franke [mailto:jornfranke@gmail.com]
> Sent: Tuesday, July 25, 2017 8:31 AM
> To: Adaryl Wakefield <ad...@hotmail.com>
> Cc: user@spark.apache.org
> Subject: Re: real world spark code
>
> Look for the ones that have unit and integration tests as well as a ci+reporting on code quality.
>
> All the others are just toy examples. Well should be :)
>
> On 25. Jul 2017, at 01:08, Adaryl Wakefield <ad...@hotmail.com> wrote:
>
> Anybody know of publicly available GitHub repos of real world Spark applications written in scala?
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.massstreet.net
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
>
RE: real world spark code
Posted by Adaryl Wakefield <ad...@hotmail.com>.
ci+reporting? I’ve never heard of that term before. What is that?
Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.massstreet.net<http://www.massstreet.net/>
www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba>
Twitter: @BobLovesData<http://twitter.com/BobLovesData>
From: Jörn Franke [mailto:jornfranke@gmail.com]
Sent: Tuesday, July 25, 2017 8:31 AM
To: Adaryl Wakefield <ad...@hotmail.com>
Cc: user@spark.apache.org
Subject: Re: real world spark code
Look for the ones that have unit and integration tests as well as a ci+reporting on code quality.
All the others are just toy examples. Well should be :)
On 25. Jul 2017, at 01:08, Adaryl Wakefield <ad...@hotmail.com>> wrote:
Anybody know of publicly available GitHub repos of real world Spark applications written in scala?
Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.massstreet.net<http://www.massstreet.net/>
www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba>
Twitter: @BobLovesData<http://twitter.com/BobLovesData>
Re: real world spark code
Posted by Jörn Franke <jo...@gmail.com>.
Look for the ones that have unit and integration tests as well as a ci+reporting on code quality.
All the others are just toy examples. Well should be :)
> On 25. Jul 2017, at 01:08, Adaryl Wakefield <ad...@hotmail.com> wrote:
>
> Anybody know of publicly available GitHub repos of real world Spark applications written in scala?
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.massstreet.net
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData
>
>
Re: real world spark code
Posted by Xiayun Sun <xi...@gmail.com>.
usually I look in github repos of those big name companies that I know are
actively doing machine learning.
For example, here are two spark-related repos from soundcloud:
- https://github.com/soundcloud/spark-pagerank
- https://github.com/soundcloud/cosine-lsh-join-spark
On 25 July 2017 at 06:08, Adaryl Wakefield <ad...@hotmail.com>
wrote:
> Anybody know of publicly available GitHub repos of real world Spark
> applications written in scala?
>
>
>
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
>
> www.massstreet.net
>
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData <http://twitter.com/BobLovesData>
>
>
>
>
>
--
Xiayun Sun
Home is behind, the world ahead,
and there are many paths to tread
through shadows to the edge of night,
until the stars are all alight.