You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jung <jb...@naver.com> on 2016/01/04 10:06:20 UTC
Is Spark 1.6 released?
Hi
There were Spark 1.6 jars in maven central and github.
I found it 5 days ago. But it doesn't appear on Spark website now.
May I regard Spark 1.6 zip file in github as a stable release?
Thanks
Jung
Re: Is Spark 1.6 released?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Jung,
yes Spark 1.6.0 has been released December, 28th.
The artifacts are on Maven Central:
http://repo1.maven.org/maven2/org/apache/spark/
However, the distribution is not available on dist.apache.org:
https://dist.apache.org/repos/dist/release/spark/
Let me check with the team to upload the distribution to dist.apache.org.
Regards
JB
On 01/04/2016 10:06 AM, Jung wrote:
> Hi
> There were Spark 1.6 jars in maven central and github.
> I found it 5 days ago. But it doesn't appear on Spark website now.
> May I regard Spark 1.6 zip file in github as a stable release?
>
> Thanks
> Jung
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Is Spark 1.6 released?
Posted by Michael Armbrust <mi...@databricks.com>.
>
> bq. In many cases, the current implementation of the Dataset API does not
> yet leverage the additional information it has and can be slower than RDDs.
>
>
Are the characteristics of cases above known so that users can decide which
> API to use ?
>
Lots of back to back operations aren't great yet because we serialize
deseriaize unnecessarily. For example:
https://github.com/databricks/spark-sql-perf/blob/master/src/main/scala/com/databricks/spark/sql/perf/DatasetPerformance.scala#L37
>
> For custom encoders, I did a quick search but didn't find the JIRA number.
> Can you share the JIRA number ?
>
This is probably the closest thing:
https://issues.apache.org/jira/browse/SPARK-7768
Re: Is Spark 1.6 released?
Posted by Ted Yu <yu...@gmail.com>.
Michael:
Nice blog. Very informative.
bq. In many cases, the current implementation of the Dataset API does not
yet leverage the additional information it has and can be slower than RDDs.
Are the characteristics of cases above known so that users can decide which
API to use ?
For custom encoders, I did a quick search but didn't find the JIRA number.
Can you share the JIRA number ?
Thanks
On Mon, Jan 4, 2016 at 12:06 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> I also wrote about it here:
> https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html
>
> And put together a bunch of examples here:
> https://docs.cloud.databricks.com/docs/spark/1.6/index.html
>
> On Mon, Jan 4, 2016 at 12:02 PM, Annabel Melongo <
> melongo_annabel@yahoo.com.invalid> wrote:
>
>> [1] http://spark.apache.org/releases/spark-release-1-6-0.html
>> [2] http://spark.apache.org/downloads.html
>>
>>
>>
>> On Monday, January 4, 2016 2:59 PM, "Saif.A.Ellafi@wellsfargo.com" <
>> Saif.A.Ellafi@wellsfargo.com> wrote:
>>
>>
>> Where can I read more about the dataset api on a user layer? I am failing
>> to get an API doc or understand when to use DataFrame or DataSet,
>> advantages, etc.
>>
>> Thanks,
>> Saif
>>
>> -----Original Message-----
>> From: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
>> Sent: Monday, January 04, 2016 2:01 PM
>> To: user@spark.apache.org
>> Subject: Re: Is Spark 1.6 released?
>>
>> It's now OK: Michael published and announced the release.
>>
>> Sorry for the delay.
>>
>> Regards
>> JB
>>
>> On 01/04/2016 10:06 AM, Jung wrote:
>> > Hi
>> > There were Spark 1.6 jars in maven central and github.
>> > I found it 5 days ago. But it doesn't appear on Spark website now.
>> > May I regard Spark 1.6 zip file in github as a stable release?
>> >
>> > Thanks
>> > Jung
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
>> commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>>
>
Re: Is Spark 1.6 released?
Posted by Michael Armbrust <mi...@databricks.com>.
I also wrote about it here:
https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html
And put together a bunch of examples here:
https://docs.cloud.databricks.com/docs/spark/1.6/index.html
On Mon, Jan 4, 2016 at 12:02 PM, Annabel Melongo <
melongo_annabel@yahoo.com.invalid> wrote:
> [1] http://spark.apache.org/releases/spark-release-1-6-0.html
> [2] http://spark.apache.org/downloads.html
>
>
>
> On Monday, January 4, 2016 2:59 PM, "Saif.A.Ellafi@wellsfargo.com" <
> Saif.A.Ellafi@wellsfargo.com> wrote:
>
>
> Where can I read more about the dataset api on a user layer? I am failing
> to get an API doc or understand when to use DataFrame or DataSet,
> advantages, etc.
>
> Thanks,
> Saif
>
> -----Original Message-----
> From: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
> Sent: Monday, January 04, 2016 2:01 PM
> To: user@spark.apache.org
> Subject: Re: Is Spark 1.6 released?
>
> It's now OK: Michael published and announced the release.
>
> Sorry for the delay.
>
> Regards
> JB
>
> On 01/04/2016 10:06 AM, Jung wrote:
> > Hi
> > There were Spark 1.6 jars in maven central and github.
> > I found it 5 days ago. But it doesn't appear on Spark website now.
> > May I regard Spark 1.6 zip file in github as a stable release?
> >
> > Thanks
> > Jung
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
> commands, e-mail: user-help@spark.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
>
Re: Is Spark 1.6 released?
Posted by Annabel Melongo <me...@yahoo.com.INVALID>.
[1] http://spark.apache.org/releases/spark-release-1-6-0.html[2] http://spark.apache.org/downloads.html
On Monday, January 4, 2016 2:59 PM, "Saif.A.Ellafi@wellsfargo.com" <Sa...@wellsfargo.com> wrote:
Where can I read more about the dataset api on a user layer? I am failing to get an API doc or understand when to use DataFrame or DataSet, advantages, etc.
Thanks,
Saif
-----Original Message-----
From: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
Sent: Monday, January 04, 2016 2:01 PM
To: user@spark.apache.org
Subject: Re: Is Spark 1.6 released?
It's now OK: Michael published and announced the release.
Sorry for the delay.
Regards
JB
On 01/04/2016 10:06 AM, Jung wrote:
> Hi
> There were Spark 1.6 jars in maven central and github.
> I found it 5 days ago. But it doesn't appear on Spark website now.
> May I regard Spark 1.6 zip file in github as a stable release?
>
> Thanks
> Jung
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Is Spark 1.6 released?
Posted by Ted Yu <yu...@gmail.com>.
Please refer to the following:
https://spark.apache.org/docs/latest/sql-programming-guide.html#datasets
https://spark.apache.org/docs/latest/sql-programming-guide.html#creating-datasets
https://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets
Cheers
On Mon, Jan 4, 2016 at 11:59 AM, <Sa...@wellsfargo.com> wrote:
> Where can I read more about the dataset api on a user layer? I am failing
> to get an API doc or understand when to use DataFrame or DataSet,
> advantages, etc.
>
> Thanks,
> Saif
>
> -----Original Message-----
> From: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
> Sent: Monday, January 04, 2016 2:01 PM
> To: user@spark.apache.org
> Subject: Re: Is Spark 1.6 released?
>
> It's now OK: Michael published and announced the release.
>
> Sorry for the delay.
>
> Regards
> JB
>
> On 01/04/2016 10:06 AM, Jung wrote:
> > Hi
> > There were Spark 1.6 jars in maven central and github.
> > I found it 5 days ago. But it doesn't appear on Spark website now.
> > May I regard Spark 1.6 zip file in github as a stable release?
> >
> > Thanks
> > Jung
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
> commands, e-mail: user-help@spark.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
RE: Is Spark 1.6 released?
Posted by Sa...@wellsfargo.com.
Where can I read more about the dataset api on a user layer? I am failing to get an API doc or understand when to use DataFrame or DataSet, advantages, etc.
Thanks,
Saif
-----Original Message-----
From: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
Sent: Monday, January 04, 2016 2:01 PM
To: user@spark.apache.org
Subject: Re: Is Spark 1.6 released?
It's now OK: Michael published and announced the release.
Sorry for the delay.
Regards
JB
On 01/04/2016 10:06 AM, Jung wrote:
> Hi
> There were Spark 1.6 jars in maven central and github.
> I found it 5 days ago. But it doesn't appear on Spark website now.
> May I regard Spark 1.6 zip file in github as a stable release?
>
> Thanks
> Jung
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Is Spark 1.6 released?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
It's now OK: Michael published and announced the release.
Sorry for the delay.
Regards
JB
On 01/04/2016 10:06 AM, Jung wrote:
> Hi
> There were Spark 1.6 jars in maven central and github.
> I found it 5 days ago. But it doesn't appear on Spark website now.
> May I regard Spark 1.6 zip file in github as a stable release?
>
> Thanks
> Jung
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org