You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Patrick Wendell <pw...@gmail.com> on 2014/09/12 02:12:38 UTC
Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Announcing Spark 1.1.0!
Posted by Nicholas Chammas <ni...@gmail.com>.
Nice work everybody! I'm looking forward to trying out this release!
On Thu, Sep 11, 2014 at 8:12 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
> the second release on the API-compatible 1.X line. It is Spark's
> largest release ever, with contributions from 171 developers!
>
> This release brings operational and performance improvements in Spark
> core including a new implementation of the Spark shuffle designed for
> very large scale workloads. Spark 1.1 adds significant extensions to
> the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
> JDBC server, byte code generation for fast expression evaluation, a
> public types API, JSON support, and other features and optimizations.
> MLlib introduces a new statistics library along with several new
> algorithms and optimizations. Spark 1.1 also builds out Spark's Python
> support and adds new components to the Spark Streaming module.
>
> Visit the release notes [1] to read about the new features, or
> download [2] the release today.
>
> [1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
> [2] http://spark.eu.apache.org/downloads.html
>
> NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL
> HOURS.
>
> Please e-mail me directly for any type-o's in the release notes or name
> listing.
>
> Thanks, and congratulations!
> - Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Announcing Spark 1.1.0!
Posted by Debasish Das <de...@gmail.com>.
Congratulations on the 1.1 release !
On Thu, Sep 11, 2014 at 9:08 PM, Matei Zaharia <ma...@gmail.com>
wrote:
> Thanks to everyone who contributed to implementing and testing this
> release!
>
> Matei
>
> On September 11, 2014 at 11:52:43 PM, Tim Smith (secsubs@gmail.com) wrote:
>
> Thanks for all the good work. Very excited about seeing more features and
> better stability in the framework.
>
>
> On Thu, Sep 11, 2014 at 5:12 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
>
>> I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
>> the second release on the API-compatible 1.X line. It is Spark's
>> largest release ever, with contributions from 171 developers!
>>
>> This release brings operational and performance improvements in Spark
>> core including a new implementation of the Spark shuffle designed for
>> very large scale workloads. Spark 1.1 adds significant extensions to
>> the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
>> JDBC server, byte code generation for fast expression evaluation, a
>> public types API, JSON support, and other features and optimizations.
>> MLlib introduces a new statistics library along with several new
>> algorithms and optimizations. Spark 1.1 also builds out Spark's Python
>> support and adds new components to the Spark Streaming module.
>>
>> Visit the release notes [1] to read about the new features, or
>> download [2] the release today.
>>
>> [1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
>> [2] http://spark.eu.apache.org/downloads.html
>>
>> NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL
>> HOURS.
>>
>> Please e-mail me directly for any type-o's in the release notes or name
>> listing.
>>
>> Thanks, and congratulations!
>> - Patrick
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Announcing Spark 1.1.0!
Posted by Matei Zaharia <ma...@gmail.com>.
Thanks to everyone who contributed to implementing and testing this release!
Matei
On September 11, 2014 at 11:52:43 PM, Tim Smith (secsubs@gmail.com) wrote:
Thanks for all the good work. Very excited about seeing more features and better stability in the framework.
On Thu, Sep 11, 2014 at 5:12 PM, Patrick Wendell <pw...@gmail.com> wrote:
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Announcing Spark 1.1.0!
Posted by Tim Smith <se...@gmail.com>.
Thanks for all the good work. Very excited about seeing more features and
better stability in the framework.
On Thu, Sep 11, 2014 at 5:12 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
> the second release on the API-compatible 1.X line. It is Spark's
> largest release ever, with contributions from 171 developers!
>
> This release brings operational and performance improvements in Spark
> core including a new implementation of the Spark shuffle designed for
> very large scale workloads. Spark 1.1 adds significant extensions to
> the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
> JDBC server, byte code generation for fast expression evaluation, a
> public types API, JSON support, and other features and optimizations.
> MLlib introduces a new statistics library along with several new
> algorithms and optimizations. Spark 1.1 also builds out Spark's Python
> support and adds new components to the Spark Streaming module.
>
> Visit the release notes [1] to read about the new features, or
> download [2] the release today.
>
> [1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
> [2] http://spark.eu.apache.org/downloads.html
>
> NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL
> HOURS.
>
> Please e-mail me directly for any type-o's in the release notes or name
> listing.
>
> Thanks, and congratulations!
> - Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Announcing Spark 1.1.0!
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Fri, Sep 12, 2014 at 9:12 AM, Patrick Wendell <pw...@gmail.com> wrote:
> I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
> the second release on the API-compatible 1.X line. It is Spark's
> largest release ever, with contributions from 171 developers!
>
Great, congratulations!! The release notes read great!
Seems like if I wait long enough for new Spark releases, my applications
will build themselves in the end ;-)
Tobias
Re: Announcing Spark 1.1.0!
Posted by Nicholas Chammas <ni...@gmail.com>.
Nice work everybody! I'm looking forward to trying out this release!
On Thu, Sep 11, 2014 at 8:12 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
> the second release on the API-compatible 1.X line. It is Spark's
> largest release ever, with contributions from 171 developers!
>
> This release brings operational and performance improvements in Spark
> core including a new implementation of the Spark shuffle designed for
> very large scale workloads. Spark 1.1 adds significant extensions to
> the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
> JDBC server, byte code generation for fast expression evaluation, a
> public types API, JSON support, and other features and optimizations.
> MLlib introduces a new statistics library along with several new
> algorithms and optimizations. Spark 1.1 also builds out Spark's Python
> support and adds new components to the Spark Streaming module.
>
> Visit the release notes [1] to read about the new features, or
> download [2] the release today.
>
> [1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
> [2] http://spark.eu.apache.org/downloads.html
>
> NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL
> HOURS.
>
> Please e-mail me directly for any type-o's in the release notes or name
> listing.
>
> Thanks, and congratulations!
> - Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
RE: Announcing Spark 1.1.0!
Posted by Haopu Wang <HW...@qilinsoft.com>.
Got it, thank you, Denny!
________________________________
From: Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 11:04 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
Yes, atleast for my query scenarios, I have been able to use Spark 1.1 with Hadoop 2.4 against Hadoop 2.5. Note, Hadoop 2.5 is considered a relatively minor release (http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available) where Hadoop 2.4 and 2.3 were considered more significant releases.
On September 11, 2014 at 19:22:05, Haopu Wang (hwang@qilinsoft.com) wrote:
From the web page (https://spark.apache.org/docs/latest/building-with-maven.html) which is pointed out by you, it’s saying “Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you’ll need to build Spark against the specific HDFS version in your environment.”
Did you try to read a hadoop 2.5.0 file using Spark 1.1 with hadoop 2.4?
Thanks!
________________________________
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 10:00 AM
To: Patrick Wendell; Haopu Wang; dev@spark.apache.org; user@spark.apache.org
Subject: RE: Announcing Spark 1.1.0!
Please correct me if I’m wrong but I was under the impression as per the maven repositories that it was just to stay more in sync with the various version of Hadoop. Looking at the latest documentation (https://spark.apache.org/docs/latest/building-with-maven.html), there are multiple Hadoop versions called out.
As for the potential differences in Spark, this is more about ensuring the various jars and library dependencies of the correct version of Hadoop are included so there can be proper connectivity to Hadoop from Spark vs. any differences in Spark itself. Another good reference on this topic is call out for Hadoop versions within github: https://github.com/apache/spark
HTH!
On September 11, 2014 at 18:39:10, Haopu Wang (hwang@qilinsoft.com) wrote:
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
________________________________
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Denny Lee <de...@gmail.com>.
Yes, atleast for my query scenarios, I have been able to use Spark 1.1 with Hadoop 2.4 against Hadoop 2.5. Note, Hadoop 2.5 is considered a relatively minor release (http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available) where Hadoop 2.4 and 2.3 were considered more significant releases.
On September 11, 2014 at 19:22:05, Haopu Wang (hwang@qilinsoft.com) wrote:
From the web page (https://spark.apache.org/docs/latest/building-with-maven.html) which is pointed out by you, it’s saying “Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you’ll need to build Spark against the specific HDFS version in your environment.”
Did you try to read a hadoop 2.5.0 file using Spark 1.1 with hadoop 2.4?
Thanks!
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 10:00 AM
To: Patrick Wendell; Haopu Wang; dev@spark.apache.org; user@spark.apache.org
Subject: RE: Announcing Spark 1.1.0!
Please correct me if I’m wrong but I was under the impression as per the maven repositories that it was just to stay more in sync with the various version of Hadoop. Looking at the latest documentation (https://spark.apache.org/docs/latest/building-with-maven.html), there are multiple Hadoop versions called out.
As for the potential differences in Spark, this is more about ensuring the various jars and library dependencies of the correct version of Hadoop are included so there can be proper connectivity to Hadoop from Spark vs. any differences in Spark itself. Another good reference on this topic is call out for Hadoop versions within github: https://github.com/apache/spark
HTH!
On September 11, 2014 at 18:39:10, Haopu Wang (hwang@qilinsoft.com) wrote:
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Denny Lee <de...@gmail.com>.
Yes, atleast for my query scenarios, I have been able to use Spark 1.1 with Hadoop 2.4 against Hadoop 2.5. Note, Hadoop 2.5 is considered a relatively minor release (http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available) where Hadoop 2.4 and 2.3 were considered more significant releases.
On September 11, 2014 at 19:22:05, Haopu Wang (hwang@qilinsoft.com) wrote:
From the web page (https://spark.apache.org/docs/latest/building-with-maven.html) which is pointed out by you, it’s saying “Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you’ll need to build Spark against the specific HDFS version in your environment.”
Did you try to read a hadoop 2.5.0 file using Spark 1.1 with hadoop 2.4?
Thanks!
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 10:00 AM
To: Patrick Wendell; Haopu Wang; dev@spark.apache.org; user@spark.apache.org
Subject: RE: Announcing Spark 1.1.0!
Please correct me if I’m wrong but I was under the impression as per the maven repositories that it was just to stay more in sync with the various version of Hadoop. Looking at the latest documentation (https://spark.apache.org/docs/latest/building-with-maven.html), there are multiple Hadoop versions called out.
As for the potential differences in Spark, this is more about ensuring the various jars and library dependencies of the correct version of Hadoop are included so there can be proper connectivity to Hadoop from Spark vs. any differences in Spark itself. Another good reference on this topic is call out for Hadoop versions within github: https://github.com/apache/spark
HTH!
On September 11, 2014 at 18:39:10, Haopu Wang (hwang@qilinsoft.com) wrote:
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Haopu Wang <HW...@qilinsoft.com>.
From the web page (https://spark.apache.org/docs/latest/building-with-maven.html) which is pointed out by you, it’s saying “Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you’ll need to build Spark against the specific HDFS version in your environment.”
Did you try to read a hadoop 2.5.0 file using Spark 1.1 with hadoop 2.4?
Thanks!
________________________________
From: Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 10:00 AM
To: Patrick Wendell; Haopu Wang; dev@spark.apache.org; user@spark.apache.org
Subject: RE: Announcing Spark 1.1.0!
Please correct me if I’m wrong but I was under the impression as per the maven repositories that it was just to stay more in sync with the various version of Hadoop. Looking at the latest documentation (https://spark.apache.org/docs/latest/building-with-maven.html), there are multiple Hadoop versions called out.
As for the potential differences in Spark, this is more about ensuring the various jars and library dependencies of the correct version of Hadoop are included so there can be proper connectivity to Hadoop from Spark vs. any differences in Spark itself. Another good reference on this topic is call out for Hadoop versions within github: https://github.com/apache/spark
HTH!
On September 11, 2014 at 18:39:10, Haopu Wang (hwang@qilinsoft.com) wrote:
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
________________________________
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Denny Lee <de...@gmail.com>.
Please correct me if I’m wrong but I was under the impression as per the maven repositories that it was just to stay more in sync with the various version of Hadoop. Looking at the latest documentation (https://spark.apache.org/docs/latest/building-with-maven.html), there are multiple Hadoop versions called out.
As for the potential differences in Spark, this is more about ensuring the various jars and library dependencies of the correct version of Hadoop are included so there can be proper connectivity to Hadoop from Spark vs. any differences in Spark itself. Another good reference on this topic is call out for Hadoop versions within github: https://github.com/apache/spark
HTH!
On September 11, 2014 at 18:39:10, Haopu Wang (hwang@qilinsoft.com) wrote:
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Denny Lee <de...@gmail.com>.
Please correct me if I’m wrong but I was under the impression as per the maven repositories that it was just to stay more in sync with the various version of Hadoop. Looking at the latest documentation (https://spark.apache.org/docs/latest/building-with-maven.html), there are multiple Hadoop versions called out.
As for the potential differences in Spark, this is more about ensuring the various jars and library dependencies of the correct version of Hadoop are included so there can be proper connectivity to Hadoop from Spark vs. any differences in Spark itself. Another good reference on this topic is call out for Hadoop versions within github: https://github.com/apache/spark
HTH!
On September 11, 2014 at 18:39:10, Haopu Wang (hwang@qilinsoft.com) wrote:
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
From:Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Haopu Wang <HW...@qilinsoft.com>.
Danny, thanks for the response.
I raise the question because in Spark 1.0.2, I saw one binary package for hadoop2, but in Spark 1.1.0, there are separate packages for hadoop 2.3 and 2.4.
That implies some difference in Spark according to hadoop version.
________________________________
From: Denny Lee [mailto:denny.g.lee@gmail.com]
Sent: Friday, September 12, 2014 9:35 AM
To: user@spark.apache.org; Haopu Wang; dev@spark.apache.org; Patrick Wendell
Subject: RE: Announcing Spark 1.1.0!
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Denny Lee <de...@gmail.com>.
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Denny Lee <de...@gmail.com>.
I’m not sure if I’m completely answering your question here but I’m currently working (on OSX) with Hadoop 2.5 and I used the Spark 1.1 with Hadoop 2.4 without any issues.
On September 11, 2014 at 18:11:46, Haopu Wang (hwang@qilinsoft.com) wrote:
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Announcing Spark 1.1.0!
Posted by Haopu Wang <HW...@qilinsoft.com>.
I see the binary packages include hadoop 1, 2.3 and 2.4.
Does Spark 1.1.0 support hadoop 2.5.0 at below address?
http://hadoop.apache.org/releases.html#11+August%2C+2014%3A+Release+2.5.0+available
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, September 12, 2014 8:13 AM
To: dev@spark.apache.org; user@spark.apache.org
Subject: Announcing Spark 1.1.0!
I am happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is
the second release on the API-compatible 1.X line. It is Spark's
largest release ever, with contributions from 171 developers!
This release brings operational and performance improvements in Spark
core including a new implementation of the Spark shuffle designed for
very large scale workloads. Spark 1.1 adds significant extensions to
the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a
JDBC server, byte code generation for fast expression evaluation, a
public types API, JSON support, and other features and optimizations.
MLlib introduces a new statistics library along with several new
algorithms and optimizations. Spark 1.1 also builds out Spark's Python
support and adds new components to the Spark Streaming module.
Visit the release notes [1] to read about the new features, or
download [2] the release today.
[1] http://spark.eu.apache.org/releases/spark-release-1-1-0.html
[2] http://spark.eu.apache.org/downloads.html
NOTE: SOME ASF DOWNLOAD MIRRORS WILL NOT CONTAIN THE RELEASE FOR SEVERAL HOURS.
Please e-mail me directly for any type-o's in the release notes or name listing.
Thanks, and congratulations!
- Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org