You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Upender Nimbekar <up...@gmail.com> on 2014/05/30 18:20:11 UTC
Spark 1.0.0 - Java 8
Great News ! I've been awaiting this release to start doing some coding
with Spark using Java 8. Can I run Spark 1.0 examples on a virtual host
with 16 GB ram and fair descent amount of hard disk ? Or do I reaaly need
to use a cluster of machines.
Second, are there any good exmaples of using MLIB on Spark. Please shoot me
in the right direction.
Thanks
Upender
On Fri, May 30, 2014 at 6:12 AM, Patrick Wendell <pw...@gmail.com> wrote:
> I'm thrilled to announce the availability of Spark 1.0.0! Spark 1.0.0
> is a milestone release as the first in the 1.0 line of releases,
> providing API stability for Spark's core interfaces.
>
> Spark 1.0.0 is Spark's largest release ever, with contributions from
> 117 developers. I'd like to thank everyone involved in this release -
> it was truly a community effort with fixes, features, and
> optimizations contributed from dozens of organizations.
>
> This release expands Spark's standard libraries, introducing a new SQL
> package (SparkSQL) which lets users integrate SQL queries into
> existing Spark workflows. MLlib, Spark's machine learning library, is
> expanded with sparse vector support and several new algorithms. The
> GraphX and Streaming libraries also introduce new features and
> optimizations. Spark's core engine adds support for secured YARN
> clusters, a unified tool for submitting Spark applications, and
> several performance and stability improvements. Finally, Spark adds
> support for Java 8 lambda syntax and improves coverage of the Java and
> Python API's.
>
> Those features only scratch the surface - check out the release notes here:
> http://spark.apache.org/releases/spark-release-1-0-0.html
>
> Note that since release artifacts were posted recently, certain
> mirrors may not have working downloads for a few hours.
>
> - Patrick
>
Re: Spark 1.0.0 - Java 8
Posted by Aaron Davidson <il...@gmail.com>.
Also, the Spark examples can run out of the box on a single machine, as
well as a cluster. See the "Master URLs" heading here:
http://spark.apache.org/docs/latest/submitting-applications.html#master-urls
On Fri, May 30, 2014 at 9:24 AM, Surendranauth Hiraman <
suren.hiraman@velos.io> wrote:
> With respect to virtual hosts, my team uses Vagrant/Virtualbox. We have 3
> CentOS VMs with 4 GB RAM each - 2 worker nodes and a master node.
>
> Everything works fine, though if you are using MapR, you have to make sure
> they are all on the same subnet.
>
> -Suren
>
>
>
> On Fri, May 30, 2014 at 12:20 PM, Upender Nimbekar <up...@gmail.com>
> wrote:
>
>> Great News ! I've been awaiting this release to start doing some coding
>> with Spark using Java 8. Can I run Spark 1.0 examples on a virtual host
>> with 16 GB ram and fair descent amount of hard disk ? Or do I reaaly need
>> to use a cluster of machines.
>> Second, are there any good exmaples of using MLIB on Spark. Please shoot
>> me in the right direction.
>>
>> Thanks
>> Upender
>>
>> On Fri, May 30, 2014 at 6:12 AM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>>
>>> I'm thrilled to announce the availability of Spark 1.0.0! Spark 1.0.0
>>> is a milestone release as the first in the 1.0 line of releases,
>>> providing API stability for Spark's core interfaces.
>>>
>>> Spark 1.0.0 is Spark's largest release ever, with contributions from
>>> 117 developers. I'd like to thank everyone involved in this release -
>>> it was truly a community effort with fixes, features, and
>>> optimizations contributed from dozens of organizations.
>>>
>>> This release expands Spark's standard libraries, introducing a new SQL
>>> package (SparkSQL) which lets users integrate SQL queries into
>>> existing Spark workflows. MLlib, Spark's machine learning library, is
>>> expanded with sparse vector support and several new algorithms. The
>>> GraphX and Streaming libraries also introduce new features and
>>> optimizations. Spark's core engine adds support for secured YARN
>>> clusters, a unified tool for submitting Spark applications, and
>>> several performance and stability improvements. Finally, Spark adds
>>> support for Java 8 lambda syntax and improves coverage of the Java and
>>> Python API's.
>>>
>>> Those features only scratch the surface - check out the release notes
>>> here:
>>> http://spark.apache.org/releases/spark-release-1-0-0.html
>>>
>>> Note that since release artifacts were posted recently, certain
>>> mirrors may not have working downloads for a few hours.
>>>
>>> - Patrick
>>>
>>
>>
>
>
> --
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
>
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@v <su...@sociocast.com>elos.io
> W: www.velos.io
>
>
Re: Spark 1.0.0 - Java 8
Posted by Surendranauth Hiraman <su...@velos.io>.
With respect to virtual hosts, my team uses Vagrant/Virtualbox. We have 3
CentOS VMs with 4 GB RAM each - 2 worker nodes and a master node.
Everything works fine, though if you are using MapR, you have to make sure
they are all on the same subnet.
-Suren
On Fri, May 30, 2014 at 12:20 PM, Upender Nimbekar <up...@gmail.com>
wrote:
> Great News ! I've been awaiting this release to start doing some coding
> with Spark using Java 8. Can I run Spark 1.0 examples on a virtual host
> with 16 GB ram and fair descent amount of hard disk ? Or do I reaaly need
> to use a cluster of machines.
> Second, are there any good exmaples of using MLIB on Spark. Please shoot
> me in the right direction.
>
> Thanks
> Upender
>
> On Fri, May 30, 2014 at 6:12 AM, Patrick Wendell <pw...@gmail.com>
> wrote:
>
>> I'm thrilled to announce the availability of Spark 1.0.0! Spark 1.0.0
>> is a milestone release as the first in the 1.0 line of releases,
>> providing API stability for Spark's core interfaces.
>>
>> Spark 1.0.0 is Spark's largest release ever, with contributions from
>> 117 developers. I'd like to thank everyone involved in this release -
>> it was truly a community effort with fixes, features, and
>> optimizations contributed from dozens of organizations.
>>
>> This release expands Spark's standard libraries, introducing a new SQL
>> package (SparkSQL) which lets users integrate SQL queries into
>> existing Spark workflows. MLlib, Spark's machine learning library, is
>> expanded with sparse vector support and several new algorithms. The
>> GraphX and Streaming libraries also introduce new features and
>> optimizations. Spark's core engine adds support for secured YARN
>> clusters, a unified tool for submitting Spark applications, and
>> several performance and stability improvements. Finally, Spark adds
>> support for Java 8 lambda syntax and improves coverage of the Java and
>> Python API's.
>>
>> Those features only scratch the surface - check out the release notes
>> here:
>> http://spark.apache.org/releases/spark-release-1-0-0.html
>>
>> Note that since release artifacts were posted recently, certain
>> mirrors may not have working downloads for a few hours.
>>
>> - Patrick
>>
>
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: Spark 1.0.0 - Java 8
Posted by Surendranauth Hiraman <su...@velos.io>.
With respect to virtual hosts, my team uses Vagrant/Virtualbox. We have 3
CentOS VMs with 4 GB RAM each - 2 worker nodes and a master node.
Everything works fine, though if you are using MapR, you have to make sure
they are all on the same subnet.
-Suren
On Fri, May 30, 2014 at 12:20 PM, Upender Nimbekar <up...@gmail.com>
wrote:
> Great News ! I've been awaiting this release to start doing some coding
> with Spark using Java 8. Can I run Spark 1.0 examples on a virtual host
> with 16 GB ram and fair descent amount of hard disk ? Or do I reaaly need
> to use a cluster of machines.
> Second, are there any good exmaples of using MLIB on Spark. Please shoot
> me in the right direction.
>
> Thanks
> Upender
>
> On Fri, May 30, 2014 at 6:12 AM, Patrick Wendell <pw...@gmail.com>
> wrote:
>
>> I'm thrilled to announce the availability of Spark 1.0.0! Spark 1.0.0
>> is a milestone release as the first in the 1.0 line of releases,
>> providing API stability for Spark's core interfaces.
>>
>> Spark 1.0.0 is Spark's largest release ever, with contributions from
>> 117 developers. I'd like to thank everyone involved in this release -
>> it was truly a community effort with fixes, features, and
>> optimizations contributed from dozens of organizations.
>>
>> This release expands Spark's standard libraries, introducing a new SQL
>> package (SparkSQL) which lets users integrate SQL queries into
>> existing Spark workflows. MLlib, Spark's machine learning library, is
>> expanded with sparse vector support and several new algorithms. The
>> GraphX and Streaming libraries also introduce new features and
>> optimizations. Spark's core engine adds support for secured YARN
>> clusters, a unified tool for submitting Spark applications, and
>> several performance and stability improvements. Finally, Spark adds
>> support for Java 8 lambda syntax and improves coverage of the Java and
>> Python API's.
>>
>> Those features only scratch the surface - check out the release notes
>> here:
>> http://spark.apache.org/releases/spark-release-1-0-0.html
>>
>> Note that since release artifacts were posted recently, certain
>> mirrors may not have working downloads for a few hours.
>>
>> - Patrick
>>
>
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io