You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Reynold Xin <rx...@apache.org> on 2016/05/18 05:40:51 UTC

[vote] Apache Spark 2.0.0-preview release (rc1)

Hi,

In the past the Apache Spark community have created preview packages (not
official releases) and used those as opportunities to ask community members
to test the upcoming versions of Apache Spark. Several people in the Apache
community have suggested we conduct votes for these preview packages and
turn them into formal releases by the Apache foundation's standard. Preview
releases are not meant to be functional, i.e. they can and highly likely
will contain critical bugs or documentation errors, but we will be able to
post them to the project's website to get wider feedback. They should
satisfy the legal requirements of Apache's release policy (
http://www.apache.org/dev/release.html) such as having proper licenses.


Please vote on releasing the following candidate as Apache Spark version
2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.0.0-preview
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is 2.0.0-preview
(8f5a04b6299e3a47aca13cbb40e72344c0114860)

The release files, including signatures, digests, etc. can be found at:
http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc

The documentation corresponding to this release can be found at:
http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/

The list of resolved issues are:
https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0


If you are a Spark user, you can help us test this release by taking an
existing Apache Spark workload and running on this candidate, then
reporting any regressions.

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Reynold Xin <rx...@databricks.com>.
This vote passes with 14 +1s (5 binding*) and no 0 or -1!  Thanks to
everyone who voted.  I'll start work on publishing the release.

+1:
Reynold Xin*
Sean Owen*
Ovidiu-Cristian MARCU
Krishna Sankar
Michael Armbrust*
Yin Huai
Joseph Bradley*
Xiangrui Meng*
Herman van Hövell tot Westerflier
Vishnu Prasad
Takuya UESHIN
Xiao Li
Ross Lawley
Ricardo Almeida

0: (none)

-1: (none)


On Tue, May 17, 2016 at 10:40 PM, Reynold Xin <rx...@apache.org> wrote:

> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy (
> http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
> and passes if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is 2.0.0-preview
> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>
> The release files, including signatures, digests, etc. can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The documentation corresponding to this release can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>
> The list of resolved issues are:
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>
>
> If you are a Spark user, you can help us test this release by taking an
> existing Apache Spark workload and running on this candidate, then
> reporting any regressions.
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Michael Armbrust <mi...@databricks.com>.
+1, excited for 2.0!

On Wed, May 18, 2016 at 10:06 AM, Krishna Sankar <ks...@gmail.com>
wrote:

> +1. Looks Good.
> The mllib results are in line with 1.6.1. Deprecation messages. I will
> convert to ml and test later in the day.
> Also will try GraphX exercises for our Strata London Tutorial
>
> Quick Notes:
>
>    1. pyspark env variables need to be changed
>    - IPYTHON and IPYTHON_OPTS are removed
>       - This works
>          - PYSPARK_DRIVER_PYTHON=ipython
>          PYSPARK_DRIVER_PYTHON_OPTS="notebook"
>          ~/Downloads/spark-2.0.0-preview/bin/pyspark --packages
>          com.databricks:spark-csv_2.10:1.4.0
>       2.  maven 3.3.9 is required. (I was running 3.3.3)
>    3.  Tons of interesting warnings and deprecations.
>       - The messages look descriptive and very helpful (Thanks. This will
>       help migration to 2.0, mllib -> ml et al). Will dig deeper.
>       4. Compiled OSX 10.10 (Yosemite) OK Total time: 31:28 min
>         mvn clean package -Pyarn -Phadoop-2.6 -DskipTests
>    - Spark version is 2.0.0-preview
>       - Tested pyspark, mllib (iPython 4.2.0)
>
> Cheers & Good work folks
> <k/>
>
> On Wed, May 18, 2016 at 7:28 AM, Sean Owen <so...@cloudera.com> wrote:
>
>> I think it's a good idea. Although releases have been preceded before
>> by release candidates for developers, it would be good to get a formal
>> preview/beta release ratified for public consumption ahead of a new
>> major release. Better to have a little more testing in the wild to
>> identify problems before 2.0.0 is finalized.
>>
>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>> 8, compilation and tests succeed for "-Pyarn -Phive
>> -Phive-thriftserver -Phadoop-2.6".
>>
>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>> > Hi,
>> >
>> > In the past the Apache Spark community have created preview packages
>> (not
>> > official releases) and used those as opportunities to ask community
>> members
>> > to test the upcoming versions of Apache Spark. Several people in the
>> Apache
>> > community have suggested we conduct votes for these preview packages and
>> > turn them into formal releases by the Apache foundation's standard.
>> Preview
>> > releases are not meant to be functional, i.e. they can and highly likely
>> > will contain critical bugs or documentation errors, but we will be able
>> to
>> > post them to the project's website to get wider feedback. They should
>> > satisfy the legal requirements of Apache's release policy
>> > (http://www.apache.org/dev/release.html) such as having proper
>> licenses.
>> >
>> >
>> > Please vote on releasing the following candidate as Apache Spark version
>> > 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
>> PDT
>> > and passes if a majority of at least 3 +1 PMC votes are cast.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.0.0-preview
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is 2.0.0-preview
>> > (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> >
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>> >
>> > Release artifacts are signed with the following key:
>> > https://people.apache.org/keys/committer/pwendell.asc
>> >
>> > The documentation corresponding to this release can be found at:
>> >
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>> >
>> > The list of resolved issues are:
>> >
>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>> >
>> >
>> > If you are a Spark user, you can help us test this release by taking an
>> > existing Apache Spark workload and running on this candidate, then
>> reporting
>> > any regressions.
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Krishna Sankar <ks...@gmail.com>.
+1. Looks Good.
The mllib results are in line with 1.6.1. Deprecation messages. I will
convert to ml and test later in the day.
Also will try GraphX exercises for our Strata London Tutorial

Quick Notes:

   1. pyspark env variables need to be changed
   - IPYTHON and IPYTHON_OPTS are removed
      - This works
         - PYSPARK_DRIVER_PYTHON=ipython
         PYSPARK_DRIVER_PYTHON_OPTS="notebook"
         ~/Downloads/spark-2.0.0-preview/bin/pyspark --packages
         com.databricks:spark-csv_2.10:1.4.0
      2.  maven 3.3.9 is required. (I was running 3.3.3)
   3.  Tons of interesting warnings and deprecations.
      - The messages look descriptive and very helpful (Thanks. This will
      help migration to 2.0, mllib -> ml et al). Will dig deeper.
      4. Compiled OSX 10.10 (Yosemite) OK Total time: 31:28 min
        mvn clean package -Pyarn -Phadoop-2.6 -DskipTests
   - Spark version is 2.0.0-preview
      - Tested pyspark, mllib (iPython 4.2.0)

Cheers & Good work folks
<k/>

On Wed, May 18, 2016 at 7:28 AM, Sean Owen <so...@cloudera.com> wrote:

> I think it's a good idea. Although releases have been preceded before
> by release candidates for developers, it would be good to get a formal
> preview/beta release ratified for public consumption ahead of a new
> major release. Better to have a little more testing in the wild to
> identify problems before 2.0.0 is finalized.
>
> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
> 8, compilation and tests succeed for "-Pyarn -Phive
> -Phive-thriftserver -Phadoop-2.6".
>
> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
> > Hi,
> >
> > In the past the Apache Spark community have created preview packages (not
> > official releases) and used those as opportunities to ask community
> members
> > to test the upcoming versions of Apache Spark. Several people in the
> Apache
> > community have suggested we conduct votes for these preview packages and
> > turn them into formal releases by the Apache foundation's standard.
> Preview
> > releases are not meant to be functional, i.e. they can and highly likely
> > will contain critical bugs or documentation errors, but we will be able
> to
> > post them to the project's website to get wider feedback. They should
> > satisfy the legal requirements of Apache's release policy
> > (http://www.apache.org/dev/release.html) such as having proper licenses.
> >
> >
> > Please vote on releasing the following candidate as Apache Spark version
> > 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
> PDT
> > and passes if a majority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Spark 2.0.0-preview
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is 2.0.0-preview
> > (8f5a04b6299e3a47aca13cbb40e72344c0114860)
> >
> > The release files, including signatures, digests, etc. can be found at:
> > http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pwendell.asc
> >
> > The documentation corresponding to this release can be found at:
> >
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
> >
> > The list of resolved issues are:
> >
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
> >
> >
> > If you are a Spark user, you can help us test this release by taking an
> > existing Apache Spark workload and running on this candidate, then
> reporting
> > any regressions.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Jeff Zhang <zj...@gmail.com>.
@Xiao,

It is tracked in SPARK-15345
<https://issues.apache.org/jira/browse/SPARK-15345>

On Fri, May 20, 2016 at 4:20 AM, Xiao Li <ga...@gmail.com> wrote:

> -1
>
> Unable to use Hive meta-store in pyspark shell. Tried both HiveContext and
> SparkSession. Both failed. It always uses in-memory catalog. Anybody else
> hit the same issue?
>
>
> Method 1: SparkSession
>
> >>> from pyspark.sql import SparkSession
>
> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>
> >>>
>
> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>
> DataFrame[]
>
> >>> spark.sql("LOAD DATA LOCAL INPATH
> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>
> Traceback (most recent call last):
>
>   File "<stdin>", line 1, in <module>
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
> line 494, in sql
>
>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
> line 933, in __call__
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
> line 57, in deco
>
>     return f(*a, **kw)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
> line 312, in get_return_value
>
> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>
> : java.lang.UnsupportedOperationException: loadTable is not implemented
>
> at
> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>
> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>
> at py4j.Gateway.invoke(Gateway.java:280)
>
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>
> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Method 2: Using HiveContext:
>
> >>> from pyspark.sql import HiveContext
>
> >>> sqlContext = HiveContext(sc)
>
> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
> STRING)")
>
> DataFrame[]
>
> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>
> Traceback (most recent call last):
>
>   File "<stdin>", line 1, in <module>
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
> line 346, in sql
>
>     return self.sparkSession.sql(sqlQuery)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
> line 494, in sql
>
>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
> line 933, in __call__
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
> line 57, in deco
>
>     return f(*a, **kw)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
> line 312, in get_return_value
>
> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>
> : java.lang.UnsupportedOperationException: loadTable is not implemented
>
> at
> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>
> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>
> at py4j.Gateway.invoke(Gateway.java:280)
>
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>
> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
> hvanhovell@questtec.nl>:
>
>> +1
>>
>>
>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>
>>> +1
>>>
>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>>> wrote:
>>>>
>>>>> Hi Ovidiu-Cristian ,
>>>>>
>>>>> The best source of truth is change the filter with target version to
>>>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>
>>>>>> Yes, I can filter..
>>>>>> Did that and for example:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>
>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>
>>>>>> Keep up the good work!
>>>>>>
>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>>
>>>>>> You can find that by changing the filter to target version = 2.0.0.
>>>>>> Cheers.
>>>>>>
>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>
>>>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>>>> known issue you plan to stay with this release?
>>>>>>>
>>>>>>> with
>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>
>>>>>>> mvn -version
>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>> Java home:
>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family:
>>>>>>> “mac"
>>>>>>>
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] Spark Project Parent POM ........................... SUCCESS
>>>>>>> [  2.635 s]
>>>>>>> [INFO] Spark Project Tags ................................. SUCCESS
>>>>>>> [  1.896 s]
>>>>>>> [INFO] Spark Project Sketch ............................... SUCCESS
>>>>>>> [  2.560 s]
>>>>>>> [INFO] Spark Project Networking ........................... SUCCESS
>>>>>>> [  6.533 s]
>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS
>>>>>>> [  4.176 s]
>>>>>>> [INFO] Spark Project Unsafe ............................... SUCCESS
>>>>>>> [  4.809 s]
>>>>>>> [INFO] Spark Project Launcher ............................. SUCCESS
>>>>>>> [  6.242 s]
>>>>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>>>>> [01:20 min]
>>>>>>> [INFO] Spark Project GraphX ............................... SUCCESS
>>>>>>> [  9.148 s]
>>>>>>> [INFO] Spark Project Streaming ............................ SUCCESS
>>>>>>> [ 22.760 s]
>>>>>>> [INFO] Spark Project Catalyst ............................. SUCCESS
>>>>>>> [ 50.783 s]
>>>>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>>>>> [01:05 min]
>>>>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS
>>>>>>> [  4.281 s]
>>>>>>> [INFO] Spark Project ML Library ........................... SUCCESS
>>>>>>> [ 54.537 s]
>>>>>>> [INFO] Spark Project Tools ................................ SUCCESS
>>>>>>> [  0.747 s]
>>>>>>> [INFO] Spark Project Hive ................................. SUCCESS
>>>>>>> [ 33.032 s]
>>>>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS
>>>>>>> [  3.198 s]
>>>>>>> [INFO] Spark Project REPL ................................. SUCCESS
>>>>>>> [  3.573 s]
>>>>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
>>>>>>> [  4.617 s]
>>>>>>> [INFO] Spark Project YARN ................................. SUCCESS
>>>>>>> [  7.321 s]
>>>>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS
>>>>>>> [ 16.496 s]
>>>>>>> [INFO] Spark Project Assembly ............................. SUCCESS
>>>>>>> [  2.300 s]
>>>>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS
>>>>>>> [  4.219 s]
>>>>>>> [INFO] Spark Project External Flume ....................... SUCCESS
>>>>>>> [  6.987 s]
>>>>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS
>>>>>>> [  1.465 s]
>>>>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS
>>>>>>> [  6.891 s]
>>>>>>> [INFO] Spark Project Examples ............................. SUCCESS
>>>>>>> [ 13.465 s]
>>>>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS
>>>>>>> [  2.815 s]
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD SUCCESS
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 07:04 min
>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>
>>>>>>> I think it's a good idea. Although releases have been preceded before
>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>> formal
>>>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>>>> major release. Better to have a little more testing in the wild to
>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>
>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> In the past the Apache Spark community have created preview packages
>>>>>>> (not
>>>>>>> official releases) and used those as opportunities to ask community
>>>>>>> members
>>>>>>> to test the upcoming versions of Apache Spark. Several people in the
>>>>>>> Apache
>>>>>>> community have suggested we conduct votes for these preview packages
>>>>>>> and
>>>>>>> turn them into formal releases by the Apache foundation's standard.
>>>>>>> Preview
>>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>>> likely
>>>>>>> will contain critical bugs or documentation errors, but we will be
>>>>>>> able to
>>>>>>> post them to the project's website to get wider feedback. They should
>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>> licenses.
>>>>>>>
>>>>>>>
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>> version
>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00
>>>>>>> PM PDT
>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>>
>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>>> at:
>>>>>>>
>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>
>>>>>>> Release artifacts are signed with the following key:
>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>
>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>
>>>>>>> The list of resolved issues are:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>
>>>>>>>
>>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>>> an
>>>>>>> existing Apache Spark workload and running on this candidate, then
>>>>>>> reporting
>>>>>>> any regressions.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>
>


-- 
Best Regards

Jeff Zhang

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Ricardo Almeida <ri...@actnowib.com>.
+1

----
Ricardo Almeida

On 20 May 2016 at 18:33, Mark Hamstra <ma...@clearstorydata.com> wrote:

> This is isn't yet a release candidate since, as Reynold mention in his
> opening post, preview releases are "not meant to be functional, i.e. they
> can and highly likely will contain critical bugs or documentation errors."
>  Once we're at the point where we expect there not to be such bugs and
> errors, then the release candidates will start.
>
> On Fri, May 20, 2016 at 4:40 AM, Ross Lawley <ro...@gmail.com>
> wrote:
>
>> +1 Having an rc1 would help me get stable feedback on using my library
>> with Spark, compared to relying on 2.0.0-SNAPSHOT.
>>
>>
>> On Fri, 20 May 2016 at 05:57 Xiao Li <ga...@gmail.com> wrote:
>>
>>> Changed my vote to +1. Thanks!
>>>
>>> 2016-05-19 13:28 GMT-07:00 Xiao Li <ga...@gmail.com>:
>>>
>>>> Will do. Thanks!
>>>>
>>>> 2016-05-19 13:26 GMT-07:00 Reynold Xin <rx...@databricks.com>:
>>>>
>>>>> Xiao thanks for posting. Please file a bug in JIRA. Again as I said in
>>>>> the email this is not meant to be a functional release and will contain
>>>>> bugs.
>>>>>
>>>>> On Thu, May 19, 2016 at 1:20 PM, Xiao Li <ga...@gmail.com> wrote:
>>>>>
>>>>>> -1
>>>>>>
>>>>>> Unable to use Hive meta-store in pyspark shell. Tried both
>>>>>> HiveContext and SparkSession. Both failed. It always uses in-memory
>>>>>> catalog. Anybody else hit the same issue?
>>>>>>
>>>>>>
>>>>>> Method 1: SparkSession
>>>>>>
>>>>>> >>> from pyspark.sql import SparkSession
>>>>>>
>>>>>> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>>>>>>
>>>>>> >>>
>>>>>>
>>>>>> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
>>>>>> STRING)")
>>>>>>
>>>>>> DataFrame[]
>>>>>>
>>>>>> >>> spark.sql("LOAD DATA LOCAL INPATH
>>>>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>
>>>>>>   File "<stdin>", line 1, in <module>
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>>>>> line 494, in sql
>>>>>>
>>>>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>>>>> line 933, in __call__
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>>>>> line 57, in deco
>>>>>>
>>>>>>     return f(*a, **kw)
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>>>>> line 312, in get_return_value
>>>>>>
>>>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>>>>
>>>>>> : java.lang.UnsupportedOperationException: loadTable is not
>>>>>> implemented
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>>>>
>>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>>>>
>>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>>>>
>>>>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>>>>
>>>>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>>>>
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>
>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>>>>
>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>>>>
>>>>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>>>>
>>>>>> at
>>>>>> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>>>>
>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>
>>>>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> Method 2: Using HiveContext:
>>>>>>
>>>>>> >>> from pyspark.sql import HiveContext
>>>>>>
>>>>>> >>> sqlContext = HiveContext(sc)
>>>>>>
>>>>>> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
>>>>>> STRING)")
>>>>>>
>>>>>> DataFrame[]
>>>>>>
>>>>>> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
>>>>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>
>>>>>>   File "<stdin>", line 1, in <module>
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
>>>>>> line 346, in sql
>>>>>>
>>>>>>     return self.sparkSession.sql(sqlQuery)
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>>>>> line 494, in sql
>>>>>>
>>>>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>>>>> line 933, in __call__
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>>>>> line 57, in deco
>>>>>>
>>>>>>     return f(*a, **kw)
>>>>>>
>>>>>>   File
>>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>>>>> line 312, in get_return_value
>>>>>>
>>>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>>>>
>>>>>> : java.lang.UnsupportedOperationException: loadTable is not
>>>>>> implemented
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>>>>
>>>>>> at
>>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>>>>
>>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>>>>
>>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>>>>
>>>>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>>>>
>>>>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>>>>
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>
>>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>>>>
>>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>>>>
>>>>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>>>>
>>>>>> at
>>>>>> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>>>>
>>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>
>>>>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
>>>>>> hvanhovell@questtec.nl>:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>>
>>>>>>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <
>>>>>>>> joseph@databricks.com> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rxin@databricks.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hi Ovidiu-Cristian ,
>>>>>>>>>>
>>>>>>>>>> The best source of truth is change the filter with target version
>>>>>>>>>> to 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as
>>>>>>>>>> we get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, I can filter..
>>>>>>>>>>> Did that and for example:
>>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>>>>>>
>>>>>>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>>>>>>
>>>>>>>>>>> Keep up the good work!
>>>>>>>>>>>
>>>>>>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> You can find that by changing the filter to target version =
>>>>>>>>>>> 2.0.0. Cheers.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> +1 Great, I see the list of resolved issues, do you have a list
>>>>>>>>>>>> of known issue you plan to stay with this release?
>>>>>>>>>>>>
>>>>>>>>>>>> with
>>>>>>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>>>>>>
>>>>>>>>>>>> mvn -version
>>>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>>>>>>> Java home:
>>>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64",
>>>>>>>>>>>> family: “mac"
>>>>>>>>>>>>
>>>>>>>>>>>> [INFO] Reactor Summary:
>>>>>>>>>>>> [INFO]
>>>>>>>>>>>> [INFO] Spark Project Parent POM ...........................
>>>>>>>>>>>> SUCCESS [  2.635 s]
>>>>>>>>>>>> [INFO] Spark Project Tags .................................
>>>>>>>>>>>> SUCCESS [  1.896 s]
>>>>>>>>>>>> [INFO] Spark Project Sketch ...............................
>>>>>>>>>>>> SUCCESS [  2.560 s]
>>>>>>>>>>>> [INFO] Spark Project Networking ...........................
>>>>>>>>>>>> SUCCESS [  6.533 s]
>>>>>>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............
>>>>>>>>>>>> SUCCESS [  4.176 s]
>>>>>>>>>>>> [INFO] Spark Project Unsafe ...............................
>>>>>>>>>>>> SUCCESS [  4.809 s]
>>>>>>>>>>>> [INFO] Spark Project Launcher .............................
>>>>>>>>>>>> SUCCESS [  6.242 s]
>>>>>>>>>>>> [INFO] Spark Project Core .................................
>>>>>>>>>>>> SUCCESS [01:20 min]
>>>>>>>>>>>> [INFO] Spark Project GraphX ...............................
>>>>>>>>>>>> SUCCESS [  9.148 s]
>>>>>>>>>>>> [INFO] Spark Project Streaming ............................
>>>>>>>>>>>> SUCCESS [ 22.760 s]
>>>>>>>>>>>> [INFO] Spark Project Catalyst .............................
>>>>>>>>>>>> SUCCESS [ 50.783 s]
>>>>>>>>>>>> [INFO] Spark Project SQL ..................................
>>>>>>>>>>>> SUCCESS [01:05 min]
>>>>>>>>>>>> [INFO] Spark Project ML Local Library .....................
>>>>>>>>>>>> SUCCESS [  4.281 s]
>>>>>>>>>>>> [INFO] Spark Project ML Library ...........................
>>>>>>>>>>>> SUCCESS [ 54.537 s]
>>>>>>>>>>>> [INFO] Spark Project Tools ................................
>>>>>>>>>>>> SUCCESS [  0.747 s]
>>>>>>>>>>>> [INFO] Spark Project Hive .................................
>>>>>>>>>>>> SUCCESS [ 33.032 s]
>>>>>>>>>>>> [INFO] Spark Project HiveContext Compatibility ............
>>>>>>>>>>>> SUCCESS [  3.198 s]
>>>>>>>>>>>> [INFO] Spark Project REPL .................................
>>>>>>>>>>>> SUCCESS [  3.573 s]
>>>>>>>>>>>> [INFO] Spark Project YARN Shuffle Service .................
>>>>>>>>>>>> SUCCESS [  4.617 s]
>>>>>>>>>>>> [INFO] Spark Project YARN .................................
>>>>>>>>>>>> SUCCESS [  7.321 s]
>>>>>>>>>>>> [INFO] Spark Project Hive Thrift Server ...................
>>>>>>>>>>>> SUCCESS [ 16.496 s]
>>>>>>>>>>>> [INFO] Spark Project Assembly .............................
>>>>>>>>>>>> SUCCESS [  2.300 s]
>>>>>>>>>>>> [INFO] Spark Project External Flume Sink ..................
>>>>>>>>>>>> SUCCESS [  4.219 s]
>>>>>>>>>>>> [INFO] Spark Project External Flume .......................
>>>>>>>>>>>> SUCCESS [  6.987 s]
>>>>>>>>>>>> [INFO] Spark Project External Flume Assembly ..............
>>>>>>>>>>>> SUCCESS [  1.465 s]
>>>>>>>>>>>> [INFO] Spark Integration for Kafka 0.8 ....................
>>>>>>>>>>>> SUCCESS [  6.891 s]
>>>>>>>>>>>> [INFO] Spark Project Examples .............................
>>>>>>>>>>>> SUCCESS [ 13.465 s]
>>>>>>>>>>>> [INFO] Spark Project External Kafka Assembly ..............
>>>>>>>>>>>> SUCCESS [  2.815 s]
>>>>>>>>>>>> [INFO]
>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>> [INFO] BUILD SUCCESS
>>>>>>>>>>>> [INFO]
>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>> [INFO] Total time: 07:04 min
>>>>>>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>>>>>>> [INFO]
>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I think it's a good idea. Although releases have been preceded
>>>>>>>>>>>> before
>>>>>>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>>>>>>> formal
>>>>>>>>>>>> preview/beta release ratified for public consumption ahead of a
>>>>>>>>>>>> new
>>>>>>>>>>>> major release. Better to have a little more testing in the wild
>>>>>>>>>>>> to
>>>>>>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>>>>>>
>>>>>>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 +
>>>>>>>>>>>> Java
>>>>>>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> In the past the Apache Spark community have created preview
>>>>>>>>>>>> packages (not
>>>>>>>>>>>> official releases) and used those as opportunities to ask
>>>>>>>>>>>> community members
>>>>>>>>>>>> to test the upcoming versions of Apache Spark. Several people
>>>>>>>>>>>> in the Apache
>>>>>>>>>>>> community have suggested we conduct votes for these preview
>>>>>>>>>>>> packages and
>>>>>>>>>>>> turn them into formal releases by the Apache foundation's
>>>>>>>>>>>> standard. Preview
>>>>>>>>>>>> releases are not meant to be functional, i.e. they can and
>>>>>>>>>>>> highly likely
>>>>>>>>>>>> will contain critical bugs or documentation errors, but we will
>>>>>>>>>>>> be able to
>>>>>>>>>>>> post them to the project's website to get wider feedback. They
>>>>>>>>>>>> should
>>>>>>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>>>>>>> licenses.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Please vote on releasing the following candidate as Apache
>>>>>>>>>>>> Spark version
>>>>>>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at
>>>>>>>>>>>> 11:00 PM PDT
>>>>>>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>>>>>>
>>>>>>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>>>>>>
>>>>>>>>>>>> To learn more about Apache Spark, please see
>>>>>>>>>>>> http://spark.apache.org/
>>>>>>>>>>>>
>>>>>>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>>>>>>
>>>>>>>>>>>> The release files, including signatures, digests, etc. can be
>>>>>>>>>>>> found at:
>>>>>>>>>>>>
>>>>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>>>>>>
>>>>>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>>>>>>
>>>>>>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>>>>>>
>>>>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>>>>>>
>>>>>>>>>>>> The list of resolved issues are:
>>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If you are a Spark user, you can help us test this release by
>>>>>>>>>>>> taking an
>>>>>>>>>>>> existing Apache Spark workload and running on this candidate,
>>>>>>>>>>>> then reporting
>>>>>>>>>>>> any regressions.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Mark Hamstra <ma...@clearstorydata.com>.
This is isn't yet a release candidate since, as Reynold mention in his
opening post, preview releases are "not meant to be functional, i.e. they
can and highly likely will contain critical bugs or documentation errors."
 Once we're at the point where we expect there not to be such bugs and
errors, then the release candidates will start.

On Fri, May 20, 2016 at 4:40 AM, Ross Lawley <ro...@gmail.com> wrote:

> +1 Having an rc1 would help me get stable feedback on using my library
> with Spark, compared to relying on 2.0.0-SNAPSHOT.
>
>
> On Fri, 20 May 2016 at 05:57 Xiao Li <ga...@gmail.com> wrote:
>
>> Changed my vote to +1. Thanks!
>>
>> 2016-05-19 13:28 GMT-07:00 Xiao Li <ga...@gmail.com>:
>>
>>> Will do. Thanks!
>>>
>>> 2016-05-19 13:26 GMT-07:00 Reynold Xin <rx...@databricks.com>:
>>>
>>>> Xiao thanks for posting. Please file a bug in JIRA. Again as I said in
>>>> the email this is not meant to be a functional release and will contain
>>>> bugs.
>>>>
>>>> On Thu, May 19, 2016 at 1:20 PM, Xiao Li <ga...@gmail.com> wrote:
>>>>
>>>>> -1
>>>>>
>>>>> Unable to use Hive meta-store in pyspark shell. Tried both HiveContext
>>>>> and SparkSession. Both failed. It always uses in-memory catalog. Anybody
>>>>> else hit the same issue?
>>>>>
>>>>>
>>>>> Method 1: SparkSession
>>>>>
>>>>> >>> from pyspark.sql import SparkSession
>>>>>
>>>>> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>>>>>
>>>>> >>>
>>>>>
>>>>> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>>>>>
>>>>> DataFrame[]
>>>>>
>>>>> >>> spark.sql("LOAD DATA LOCAL INPATH
>>>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>>>
>>>>> Traceback (most recent call last):
>>>>>
>>>>>   File "<stdin>", line 1, in <module>
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>>>> line 494, in sql
>>>>>
>>>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>>>> line 933, in __call__
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>>>> line 57, in deco
>>>>>
>>>>>     return f(*a, **kw)
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>>>> line 312, in get_return_value
>>>>>
>>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>>>
>>>>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>>>
>>>>> at
>>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>>>
>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>>>
>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>>>
>>>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>>>
>>>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>
>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>>>
>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>>>
>>>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>>>
>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>>>
>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>
>>>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> Method 2: Using HiveContext:
>>>>>
>>>>> >>> from pyspark.sql import HiveContext
>>>>>
>>>>> >>> sqlContext = HiveContext(sc)
>>>>>
>>>>> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
>>>>> STRING)")
>>>>>
>>>>> DataFrame[]
>>>>>
>>>>> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
>>>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>>>
>>>>> Traceback (most recent call last):
>>>>>
>>>>>   File "<stdin>", line 1, in <module>
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
>>>>> line 346, in sql
>>>>>
>>>>>     return self.sparkSession.sql(sqlQuery)
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>>>> line 494, in sql
>>>>>
>>>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>>>> line 933, in __call__
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>>>> line 57, in deco
>>>>>
>>>>>     return f(*a, **kw)
>>>>>
>>>>>   File
>>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>>>> line 312, in get_return_value
>>>>>
>>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>>>
>>>>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>>>
>>>>> at
>>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>>>
>>>>> at
>>>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>>>
>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>>>
>>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>>>
>>>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>>>
>>>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>
>>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>>>
>>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>>>
>>>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>>>
>>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>>>
>>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>
>>>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
>>>>> hvanhovell@questtec.nl>:
>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <
>>>>>>> joseph@databricks.com> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Ovidiu-Cristian ,
>>>>>>>>>
>>>>>>>>> The best source of truth is change the filter with target version
>>>>>>>>> to 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as
>>>>>>>>> we get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, I can filter..
>>>>>>>>>> Did that and for example:
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>>>>>
>>>>>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>>>>>
>>>>>>>>>> Keep up the good work!
>>>>>>>>>>
>>>>>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> You can find that by changing the filter to target version =
>>>>>>>>>> 2.0.0. Cheers.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1 Great, I see the list of resolved issues, do you have a list
>>>>>>>>>>> of known issue you plan to stay with this release?
>>>>>>>>>>>
>>>>>>>>>>> with
>>>>>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>>>>>
>>>>>>>>>>> mvn -version
>>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>>>>>> Java home:
>>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family:
>>>>>>>>>>> “mac"
>>>>>>>>>>>
>>>>>>>>>>> [INFO] Reactor Summary:
>>>>>>>>>>> [INFO]
>>>>>>>>>>> [INFO] Spark Project Parent POM ...........................
>>>>>>>>>>> SUCCESS [  2.635 s]
>>>>>>>>>>> [INFO] Spark Project Tags .................................
>>>>>>>>>>> SUCCESS [  1.896 s]
>>>>>>>>>>> [INFO] Spark Project Sketch ...............................
>>>>>>>>>>> SUCCESS [  2.560 s]
>>>>>>>>>>> [INFO] Spark Project Networking ...........................
>>>>>>>>>>> SUCCESS [  6.533 s]
>>>>>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............
>>>>>>>>>>> SUCCESS [  4.176 s]
>>>>>>>>>>> [INFO] Spark Project Unsafe ...............................
>>>>>>>>>>> SUCCESS [  4.809 s]
>>>>>>>>>>> [INFO] Spark Project Launcher .............................
>>>>>>>>>>> SUCCESS [  6.242 s]
>>>>>>>>>>> [INFO] Spark Project Core .................................
>>>>>>>>>>> SUCCESS [01:20 min]
>>>>>>>>>>> [INFO] Spark Project GraphX ...............................
>>>>>>>>>>> SUCCESS [  9.148 s]
>>>>>>>>>>> [INFO] Spark Project Streaming ............................
>>>>>>>>>>> SUCCESS [ 22.760 s]
>>>>>>>>>>> [INFO] Spark Project Catalyst .............................
>>>>>>>>>>> SUCCESS [ 50.783 s]
>>>>>>>>>>> [INFO] Spark Project SQL ..................................
>>>>>>>>>>> SUCCESS [01:05 min]
>>>>>>>>>>> [INFO] Spark Project ML Local Library .....................
>>>>>>>>>>> SUCCESS [  4.281 s]
>>>>>>>>>>> [INFO] Spark Project ML Library ...........................
>>>>>>>>>>> SUCCESS [ 54.537 s]
>>>>>>>>>>> [INFO] Spark Project Tools ................................
>>>>>>>>>>> SUCCESS [  0.747 s]
>>>>>>>>>>> [INFO] Spark Project Hive .................................
>>>>>>>>>>> SUCCESS [ 33.032 s]
>>>>>>>>>>> [INFO] Spark Project HiveContext Compatibility ............
>>>>>>>>>>> SUCCESS [  3.198 s]
>>>>>>>>>>> [INFO] Spark Project REPL .................................
>>>>>>>>>>> SUCCESS [  3.573 s]
>>>>>>>>>>> [INFO] Spark Project YARN Shuffle Service .................
>>>>>>>>>>> SUCCESS [  4.617 s]
>>>>>>>>>>> [INFO] Spark Project YARN .................................
>>>>>>>>>>> SUCCESS [  7.321 s]
>>>>>>>>>>> [INFO] Spark Project Hive Thrift Server ...................
>>>>>>>>>>> SUCCESS [ 16.496 s]
>>>>>>>>>>> [INFO] Spark Project Assembly .............................
>>>>>>>>>>> SUCCESS [  2.300 s]
>>>>>>>>>>> [INFO] Spark Project External Flume Sink ..................
>>>>>>>>>>> SUCCESS [  4.219 s]
>>>>>>>>>>> [INFO] Spark Project External Flume .......................
>>>>>>>>>>> SUCCESS [  6.987 s]
>>>>>>>>>>> [INFO] Spark Project External Flume Assembly ..............
>>>>>>>>>>> SUCCESS [  1.465 s]
>>>>>>>>>>> [INFO] Spark Integration for Kafka 0.8 ....................
>>>>>>>>>>> SUCCESS [  6.891 s]
>>>>>>>>>>> [INFO] Spark Project Examples .............................
>>>>>>>>>>> SUCCESS [ 13.465 s]
>>>>>>>>>>> [INFO] Spark Project External Kafka Assembly ..............
>>>>>>>>>>> SUCCESS [  2.815 s]
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [INFO] BUILD SUCCESS
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [INFO] Total time: 07:04 min
>>>>>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I think it's a good idea. Although releases have been preceded
>>>>>>>>>>> before
>>>>>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>>>>>> formal
>>>>>>>>>>> preview/beta release ratified for public consumption ahead of a
>>>>>>>>>>> new
>>>>>>>>>>> major release. Better to have a little more testing in the wild
>>>>>>>>>>> to
>>>>>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>>>>>
>>>>>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 +
>>>>>>>>>>> Java
>>>>>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> In the past the Apache Spark community have created preview
>>>>>>>>>>> packages (not
>>>>>>>>>>> official releases) and used those as opportunities to ask
>>>>>>>>>>> community members
>>>>>>>>>>> to test the upcoming versions of Apache Spark. Several people in
>>>>>>>>>>> the Apache
>>>>>>>>>>> community have suggested we conduct votes for these preview
>>>>>>>>>>> packages and
>>>>>>>>>>> turn them into formal releases by the Apache foundation's
>>>>>>>>>>> standard. Preview
>>>>>>>>>>> releases are not meant to be functional, i.e. they can and
>>>>>>>>>>> highly likely
>>>>>>>>>>> will contain critical bugs or documentation errors, but we will
>>>>>>>>>>> be able to
>>>>>>>>>>> post them to the project's website to get wider feedback. They
>>>>>>>>>>> should
>>>>>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>>>>>> licenses.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>>>>>> version
>>>>>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at
>>>>>>>>>>> 11:00 PM PDT
>>>>>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>>>>>
>>>>>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>>>>>
>>>>>>>>>>> To learn more about Apache Spark, please see
>>>>>>>>>>> http://spark.apache.org/
>>>>>>>>>>>
>>>>>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>>>>>
>>>>>>>>>>> The release files, including signatures, digests, etc. can be
>>>>>>>>>>> found at:
>>>>>>>>>>>
>>>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>>>>>
>>>>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>>>>>
>>>>>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>>>>>
>>>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>>>>>
>>>>>>>>>>> The list of resolved issues are:
>>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If you are a Spark user, you can help us test this release by
>>>>>>>>>>> taking an
>>>>>>>>>>> existing Apache Spark workload and running on this candidate,
>>>>>>>>>>> then reporting
>>>>>>>>>>> any regressions.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Ross Lawley <ro...@gmail.com>.
+1 Having an rc1 would help me get stable feedback on using my library with
Spark, compared to relying on 2.0.0-SNAPSHOT.

On Fri, 20 May 2016 at 05:57 Xiao Li <ga...@gmail.com> wrote:

> Changed my vote to +1. Thanks!
>
> 2016-05-19 13:28 GMT-07:00 Xiao Li <ga...@gmail.com>:
>
>> Will do. Thanks!
>>
>> 2016-05-19 13:26 GMT-07:00 Reynold Xin <rx...@databricks.com>:
>>
>>> Xiao thanks for posting. Please file a bug in JIRA. Again as I said in
>>> the email this is not meant to be a functional release and will contain
>>> bugs.
>>>
>>> On Thu, May 19, 2016 at 1:20 PM, Xiao Li <ga...@gmail.com> wrote:
>>>
>>>> -1
>>>>
>>>> Unable to use Hive meta-store in pyspark shell. Tried both HiveContext
>>>> and SparkSession. Both failed. It always uses in-memory catalog. Anybody
>>>> else hit the same issue?
>>>>
>>>>
>>>> Method 1: SparkSession
>>>>
>>>> >>> from pyspark.sql import SparkSession
>>>>
>>>> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>>>>
>>>> >>>
>>>>
>>>> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>>>>
>>>> DataFrame[]
>>>>
>>>> >>> spark.sql("LOAD DATA LOCAL INPATH
>>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>>
>>>> Traceback (most recent call last):
>>>>
>>>>   File "<stdin>", line 1, in <module>
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>>> line 494, in sql
>>>>
>>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>>> line 933, in __call__
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>>> line 57, in deco
>>>>
>>>>     return f(*a, **kw)
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>>> line 312, in get_return_value
>>>>
>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>>
>>>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>>>
>>>> at
>>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>>
>>>> at
>>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>>
>>>> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>>
>>>> at
>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>>
>>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>>
>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>>
>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>>
>>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>>
>>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>
>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>>
>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>>
>>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>>
>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>>
>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>
>>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>>
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> Method 2: Using HiveContext:
>>>>
>>>> >>> from pyspark.sql import HiveContext
>>>>
>>>> >>> sqlContext = HiveContext(sc)
>>>>
>>>> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
>>>> STRING)")
>>>>
>>>> DataFrame[]
>>>>
>>>> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
>>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>>
>>>> Traceback (most recent call last):
>>>>
>>>>   File "<stdin>", line 1, in <module>
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
>>>> line 346, in sql
>>>>
>>>>     return self.sparkSession.sql(sqlQuery)
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>>> line 494, in sql
>>>>
>>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>>> line 933, in __call__
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>>> line 57, in deco
>>>>
>>>>     return f(*a, **kw)
>>>>
>>>>   File
>>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>>> line 312, in get_return_value
>>>>
>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>>
>>>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>>>
>>>> at
>>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>>
>>>> at
>>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>>
>>>> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>>
>>>> at
>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>>
>>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>>
>>>> at
>>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>>
>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>>
>>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>>
>>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>>
>>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>
>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>>
>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>>
>>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>>
>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>>
>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>
>>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>>
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
>>>> hvanhovell@questtec.nl>:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Ovidiu-Cristian ,
>>>>>>>>
>>>>>>>> The best source of truth is change the filter with target version
>>>>>>>> to 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as
>>>>>>>> we get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>
>>>>>>>>> Yes, I can filter..
>>>>>>>>> Did that and for example:
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>>>>
>>>>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>>>>
>>>>>>>>> Keep up the good work!
>>>>>>>>>
>>>>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>>>>>
>>>>>>>>> You can find that by changing the filter to target version =
>>>>>>>>> 2.0.0. Cheers.
>>>>>>>>>
>>>>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>>
>>>>>>>>>> +1 Great, I see the list of resolved issues, do you have a list
>>>>>>>>>> of known issue you plan to stay with this release?
>>>>>>>>>>
>>>>>>>>>> with
>>>>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>>>>
>>>>>>>>>> mvn -version
>>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>>>>> Java home:
>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family:
>>>>>>>>>> “mac"
>>>>>>>>>>
>>>>>>>>>> [INFO] Reactor Summary:
>>>>>>>>>> [INFO]
>>>>>>>>>> [INFO] Spark Project Parent POM ...........................
>>>>>>>>>> SUCCESS [  2.635 s]
>>>>>>>>>> [INFO] Spark Project Tags .................................
>>>>>>>>>> SUCCESS [  1.896 s]
>>>>>>>>>> [INFO] Spark Project Sketch ...............................
>>>>>>>>>> SUCCESS [  2.560 s]
>>>>>>>>>> [INFO] Spark Project Networking ...........................
>>>>>>>>>> SUCCESS [  6.533 s]
>>>>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............
>>>>>>>>>> SUCCESS [  4.176 s]
>>>>>>>>>> [INFO] Spark Project Unsafe ...............................
>>>>>>>>>> SUCCESS [  4.809 s]
>>>>>>>>>> [INFO] Spark Project Launcher .............................
>>>>>>>>>> SUCCESS [  6.242 s]
>>>>>>>>>> [INFO] Spark Project Core .................................
>>>>>>>>>> SUCCESS [01:20 min]
>>>>>>>>>> [INFO] Spark Project GraphX ...............................
>>>>>>>>>> SUCCESS [  9.148 s]
>>>>>>>>>> [INFO] Spark Project Streaming ............................
>>>>>>>>>> SUCCESS [ 22.760 s]
>>>>>>>>>> [INFO] Spark Project Catalyst .............................
>>>>>>>>>> SUCCESS [ 50.783 s]
>>>>>>>>>> [INFO] Spark Project SQL ..................................
>>>>>>>>>> SUCCESS [01:05 min]
>>>>>>>>>> [INFO] Spark Project ML Local Library .....................
>>>>>>>>>> SUCCESS [  4.281 s]
>>>>>>>>>> [INFO] Spark Project ML Library ...........................
>>>>>>>>>> SUCCESS [ 54.537 s]
>>>>>>>>>> [INFO] Spark Project Tools ................................
>>>>>>>>>> SUCCESS [  0.747 s]
>>>>>>>>>> [INFO] Spark Project Hive .................................
>>>>>>>>>> SUCCESS [ 33.032 s]
>>>>>>>>>> [INFO] Spark Project HiveContext Compatibility ............
>>>>>>>>>> SUCCESS [  3.198 s]
>>>>>>>>>> [INFO] Spark Project REPL .................................
>>>>>>>>>> SUCCESS [  3.573 s]
>>>>>>>>>> [INFO] Spark Project YARN Shuffle Service .................
>>>>>>>>>> SUCCESS [  4.617 s]
>>>>>>>>>> [INFO] Spark Project YARN .................................
>>>>>>>>>> SUCCESS [  7.321 s]
>>>>>>>>>> [INFO] Spark Project Hive Thrift Server ...................
>>>>>>>>>> SUCCESS [ 16.496 s]
>>>>>>>>>> [INFO] Spark Project Assembly .............................
>>>>>>>>>> SUCCESS [  2.300 s]
>>>>>>>>>> [INFO] Spark Project External Flume Sink ..................
>>>>>>>>>> SUCCESS [  4.219 s]
>>>>>>>>>> [INFO] Spark Project External Flume .......................
>>>>>>>>>> SUCCESS [  6.987 s]
>>>>>>>>>> [INFO] Spark Project External Flume Assembly ..............
>>>>>>>>>> SUCCESS [  1.465 s]
>>>>>>>>>> [INFO] Spark Integration for Kafka 0.8 ....................
>>>>>>>>>> SUCCESS [  6.891 s]
>>>>>>>>>> [INFO] Spark Project Examples .............................
>>>>>>>>>> SUCCESS [ 13.465 s]
>>>>>>>>>> [INFO] Spark Project External Kafka Assembly ..............
>>>>>>>>>> SUCCESS [  2.815 s]
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [INFO] BUILD SUCCESS
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [INFO] Total time: 07:04 min
>>>>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>> I think it's a good idea. Although releases have been preceded
>>>>>>>>>> before
>>>>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>>>>> formal
>>>>>>>>>> preview/beta release ratified for public consumption ahead of a
>>>>>>>>>> new
>>>>>>>>>> major release. Better to have a little more testing in the wild to
>>>>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>>>>
>>>>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 +
>>>>>>>>>> Java
>>>>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>>>>
>>>>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> In the past the Apache Spark community have created preview
>>>>>>>>>> packages (not
>>>>>>>>>> official releases) and used those as opportunities to ask
>>>>>>>>>> community members
>>>>>>>>>> to test the upcoming versions of Apache Spark. Several people in
>>>>>>>>>> the Apache
>>>>>>>>>> community have suggested we conduct votes for these preview
>>>>>>>>>> packages and
>>>>>>>>>> turn them into formal releases by the Apache foundation's
>>>>>>>>>> standard. Preview
>>>>>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>>>>>> likely
>>>>>>>>>> will contain critical bugs or documentation errors, but we will
>>>>>>>>>> be able to
>>>>>>>>>> post them to the project's website to get wider feedback. They
>>>>>>>>>> should
>>>>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>>>>> licenses.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>>>>> version
>>>>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at
>>>>>>>>>> 11:00 PM PDT
>>>>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>>>>
>>>>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>>>>
>>>>>>>>>> To learn more about Apache Spark, please see
>>>>>>>>>> http://spark.apache.org/
>>>>>>>>>>
>>>>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>>>>
>>>>>>>>>> The release files, including signatures, digests, etc. can be
>>>>>>>>>> found at:
>>>>>>>>>>
>>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>>>>
>>>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>>>>
>>>>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>>>>
>>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>>>>
>>>>>>>>>> The list of resolved issues are:
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If you are a Spark user, you can help us test this release by
>>>>>>>>>> taking an
>>>>>>>>>> existing Apache Spark workload and running on this candidate,
>>>>>>>>>> then reporting
>>>>>>>>>> any regressions.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Xiao Li <ga...@gmail.com>.
Changed my vote to +1. Thanks!

2016-05-19 13:28 GMT-07:00 Xiao Li <ga...@gmail.com>:

> Will do. Thanks!
>
> 2016-05-19 13:26 GMT-07:00 Reynold Xin <rx...@databricks.com>:
>
>> Xiao thanks for posting. Please file a bug in JIRA. Again as I said in
>> the email this is not meant to be a functional release and will contain
>> bugs.
>>
>> On Thu, May 19, 2016 at 1:20 PM, Xiao Li <ga...@gmail.com> wrote:
>>
>>> -1
>>>
>>> Unable to use Hive meta-store in pyspark shell. Tried both HiveContext
>>> and SparkSession. Both failed. It always uses in-memory catalog. Anybody
>>> else hit the same issue?
>>>
>>>
>>> Method 1: SparkSession
>>>
>>> >>> from pyspark.sql import SparkSession
>>>
>>> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>>>
>>> >>>
>>>
>>> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>>>
>>> DataFrame[]
>>>
>>> >>> spark.sql("LOAD DATA LOCAL INPATH
>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>
>>> Traceback (most recent call last):
>>>
>>>   File "<stdin>", line 1, in <module>
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>> line 494, in sql
>>>
>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>> line 933, in __call__
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>> line 57, in deco
>>>
>>>     return f(*a, **kw)
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>> line 312, in get_return_value
>>>
>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>
>>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>>
>>> at
>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>
>>> at
>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>
>>> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>
>>> at
>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>
>>> at
>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>
>>> at
>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>
>>> at
>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>
>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>
>>> at
>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>
>>> at
>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>
>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>
>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>
>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>
>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>
>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>
>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>
>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>
>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>
>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>
>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> Method 2: Using HiveContext:
>>>
>>> >>> from pyspark.sql import HiveContext
>>>
>>> >>> sqlContext = HiveContext(sc)
>>>
>>> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
>>> STRING)")
>>>
>>> DataFrame[]
>>>
>>> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
>>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>>
>>> Traceback (most recent call last):
>>>
>>>   File "<stdin>", line 1, in <module>
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
>>> line 346, in sql
>>>
>>>     return self.sparkSession.sql(sqlQuery)
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>>> line 494, in sql
>>>
>>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>>> line 933, in __call__
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>>> line 57, in deco
>>>
>>>     return f(*a, **kw)
>>>
>>>   File
>>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>>> line 312, in get_return_value
>>>
>>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>>
>>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>>
>>> at
>>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>>
>>> at
>>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>>
>>> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>>
>>> at
>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>>
>>> at
>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>>
>>> at
>>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>>
>>> at
>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>>
>>> at
>>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>>
>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>>
>>> at
>>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>>
>>> at
>>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>>
>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>>
>>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>>
>>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>>
>>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>
>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>>
>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>>
>>> at py4j.Gateway.invoke(Gateway.java:280)
>>>
>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>>
>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>
>>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
>>> hvanhovell@questtec.nl>:
>>>
>>>> +1
>>>>
>>>>
>>>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>>>
>>>>> +1
>>>>>
>>>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Ovidiu-Cristian ,
>>>>>>>
>>>>>>> The best source of truth is change the filter with target version to
>>>>>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>>>>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>
>>>>>>>> Yes, I can filter..
>>>>>>>> Did that and for example:
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>>>
>>>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>>>
>>>>>>>> Keep up the good work!
>>>>>>>>
>>>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>>>>
>>>>>>>> You can find that by changing the filter to target version = 2.0.0.
>>>>>>>> Cheers.
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>>
>>>>>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>>>>>> known issue you plan to stay with this release?
>>>>>>>>>
>>>>>>>>> with
>>>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>>>
>>>>>>>>> mvn -version
>>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>>>> Java home:
>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family:
>>>>>>>>> “mac"
>>>>>>>>>
>>>>>>>>> [INFO] Reactor Summary:
>>>>>>>>> [INFO]
>>>>>>>>> [INFO] Spark Project Parent POM ...........................
>>>>>>>>> SUCCESS [  2.635 s]
>>>>>>>>> [INFO] Spark Project Tags .................................
>>>>>>>>> SUCCESS [  1.896 s]
>>>>>>>>> [INFO] Spark Project Sketch ...............................
>>>>>>>>> SUCCESS [  2.560 s]
>>>>>>>>> [INFO] Spark Project Networking ...........................
>>>>>>>>> SUCCESS [  6.533 s]
>>>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............
>>>>>>>>> SUCCESS [  4.176 s]
>>>>>>>>> [INFO] Spark Project Unsafe ...............................
>>>>>>>>> SUCCESS [  4.809 s]
>>>>>>>>> [INFO] Spark Project Launcher .............................
>>>>>>>>> SUCCESS [  6.242 s]
>>>>>>>>> [INFO] Spark Project Core .................................
>>>>>>>>> SUCCESS [01:20 min]
>>>>>>>>> [INFO] Spark Project GraphX ...............................
>>>>>>>>> SUCCESS [  9.148 s]
>>>>>>>>> [INFO] Spark Project Streaming ............................
>>>>>>>>> SUCCESS [ 22.760 s]
>>>>>>>>> [INFO] Spark Project Catalyst .............................
>>>>>>>>> SUCCESS [ 50.783 s]
>>>>>>>>> [INFO] Spark Project SQL ..................................
>>>>>>>>> SUCCESS [01:05 min]
>>>>>>>>> [INFO] Spark Project ML Local Library .....................
>>>>>>>>> SUCCESS [  4.281 s]
>>>>>>>>> [INFO] Spark Project ML Library ...........................
>>>>>>>>> SUCCESS [ 54.537 s]
>>>>>>>>> [INFO] Spark Project Tools ................................
>>>>>>>>> SUCCESS [  0.747 s]
>>>>>>>>> [INFO] Spark Project Hive .................................
>>>>>>>>> SUCCESS [ 33.032 s]
>>>>>>>>> [INFO] Spark Project HiveContext Compatibility ............
>>>>>>>>> SUCCESS [  3.198 s]
>>>>>>>>> [INFO] Spark Project REPL .................................
>>>>>>>>> SUCCESS [  3.573 s]
>>>>>>>>> [INFO] Spark Project YARN Shuffle Service .................
>>>>>>>>> SUCCESS [  4.617 s]
>>>>>>>>> [INFO] Spark Project YARN .................................
>>>>>>>>> SUCCESS [  7.321 s]
>>>>>>>>> [INFO] Spark Project Hive Thrift Server ...................
>>>>>>>>> SUCCESS [ 16.496 s]
>>>>>>>>> [INFO] Spark Project Assembly .............................
>>>>>>>>> SUCCESS [  2.300 s]
>>>>>>>>> [INFO] Spark Project External Flume Sink ..................
>>>>>>>>> SUCCESS [  4.219 s]
>>>>>>>>> [INFO] Spark Project External Flume .......................
>>>>>>>>> SUCCESS [  6.987 s]
>>>>>>>>> [INFO] Spark Project External Flume Assembly ..............
>>>>>>>>> SUCCESS [  1.465 s]
>>>>>>>>> [INFO] Spark Integration for Kafka 0.8 ....................
>>>>>>>>> SUCCESS [  6.891 s]
>>>>>>>>> [INFO] Spark Project Examples .............................
>>>>>>>>> SUCCESS [ 13.465 s]
>>>>>>>>> [INFO] Spark Project External Kafka Assembly ..............
>>>>>>>>> SUCCESS [  2.815 s]
>>>>>>>>> [INFO]
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> [INFO] BUILD SUCCESS
>>>>>>>>> [INFO]
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>> [INFO] Total time: 07:04 min
>>>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>>>> [INFO]
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>>>
>>>>>>>>> I think it's a good idea. Although releases have been preceded
>>>>>>>>> before
>>>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>>>> formal
>>>>>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>>>>>> major release. Better to have a little more testing in the wild to
>>>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>>>
>>>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 +
>>>>>>>>> Java
>>>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>>>
>>>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> In the past the Apache Spark community have created preview
>>>>>>>>> packages (not
>>>>>>>>> official releases) and used those as opportunities to ask
>>>>>>>>> community members
>>>>>>>>> to test the upcoming versions of Apache Spark. Several people in
>>>>>>>>> the Apache
>>>>>>>>> community have suggested we conduct votes for these preview
>>>>>>>>> packages and
>>>>>>>>> turn them into formal releases by the Apache foundation's
>>>>>>>>> standard. Preview
>>>>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>>>>> likely
>>>>>>>>> will contain critical bugs or documentation errors, but we will be
>>>>>>>>> able to
>>>>>>>>> post them to the project's website to get wider feedback. They
>>>>>>>>> should
>>>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>>>> licenses.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>>>> version
>>>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at
>>>>>>>>> 11:00 PM PDT
>>>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>>>
>>>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>>>
>>>>>>>>> To learn more about Apache Spark, please see
>>>>>>>>> http://spark.apache.org/
>>>>>>>>>
>>>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>>>
>>>>>>>>> The release files, including signatures, digests, etc. can be
>>>>>>>>> found at:
>>>>>>>>>
>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>>>
>>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>>>
>>>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>>>
>>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>>>
>>>>>>>>> The list of resolved issues are:
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you are a Spark user, you can help us test this release by
>>>>>>>>> taking an
>>>>>>>>> existing Apache Spark workload and running on this candidate, then
>>>>>>>>> reporting
>>>>>>>>> any regressions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Xiao Li <ga...@gmail.com>.
Will do. Thanks!

2016-05-19 13:26 GMT-07:00 Reynold Xin <rx...@databricks.com>:

> Xiao thanks for posting. Please file a bug in JIRA. Again as I said in the
> email this is not meant to be a functional release and will contain bugs.
>
> On Thu, May 19, 2016 at 1:20 PM, Xiao Li <ga...@gmail.com> wrote:
>
>> -1
>>
>> Unable to use Hive meta-store in pyspark shell. Tried both HiveContext
>> and SparkSession. Both failed. It always uses in-memory catalog. Anybody
>> else hit the same issue?
>>
>>
>> Method 1: SparkSession
>>
>> >>> from pyspark.sql import SparkSession
>>
>> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>>
>> >>>
>>
>> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>>
>> DataFrame[]
>>
>> >>> spark.sql("LOAD DATA LOCAL INPATH
>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>
>> Traceback (most recent call last):
>>
>>   File "<stdin>", line 1, in <module>
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>> line 494, in sql
>>
>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>> line 933, in __call__
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>> line 57, in deco
>>
>>     return f(*a, **kw)
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>> line 312, in get_return_value
>>
>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>
>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>
>> at
>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>
>> at
>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>
>> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>
>> at
>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>
>> at
>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>
>> at
>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>
>> at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>
>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>
>> at
>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>
>> at
>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>
>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>
>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>
>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>
>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>
>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>
>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>
>> at py4j.Gateway.invoke(Gateway.java:280)
>>
>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>
>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>
>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Method 2: Using HiveContext:
>>
>> >>> from pyspark.sql import HiveContext
>>
>> >>> sqlContext = HiveContext(sc)
>>
>> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
>> STRING)")
>>
>> DataFrame[]
>>
>> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
>> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>>
>> Traceback (most recent call last):
>>
>>   File "<stdin>", line 1, in <module>
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
>> line 346, in sql
>>
>>     return self.sparkSession.sql(sqlQuery)
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
>> line 494, in sql
>>
>>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
>> line 933, in __call__
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
>> line 57, in deco
>>
>>     return f(*a, **kw)
>>
>>   File
>> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
>> line 312, in get_return_value
>>
>> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>>
>> : java.lang.UnsupportedOperationException: loadTable is not implemented
>>
>> at
>> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>>
>> at
>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>>
>> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>>
>> at
>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>>
>> at
>> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>>
>> at
>> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>>
>> at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>
>> at
>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>>
>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>>
>> at
>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>>
>> at
>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>>
>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>>
>> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>>
>> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>>
>> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>
>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>>
>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>>
>> at py4j.Gateway.invoke(Gateway.java:280)
>>
>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>>
>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>
>> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
>> hvanhovell@questtec.nl>:
>>
>>> +1
>>>
>>>
>>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>>
>>>> +1
>>>>
>>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Ovidiu-Cristian ,
>>>>>>
>>>>>> The best source of truth is change the filter with target version to
>>>>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>>>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>
>>>>>>> Yes, I can filter..
>>>>>>> Did that and for example:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>>
>>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>>
>>>>>>> Keep up the good work!
>>>>>>>
>>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>>>
>>>>>>> You can find that by changing the filter to target version = 2.0.0.
>>>>>>> Cheers.
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>>
>>>>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>>>>> known issue you plan to stay with this release?
>>>>>>>>
>>>>>>>> with
>>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>>
>>>>>>>> mvn -version
>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>>> Java home:
>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family:
>>>>>>>> “mac"
>>>>>>>>
>>>>>>>> [INFO] Reactor Summary:
>>>>>>>> [INFO]
>>>>>>>> [INFO] Spark Project Parent POM ........................... SUCCESS
>>>>>>>> [  2.635 s]
>>>>>>>> [INFO] Spark Project Tags ................................. SUCCESS
>>>>>>>> [  1.896 s]
>>>>>>>> [INFO] Spark Project Sketch ............................... SUCCESS
>>>>>>>> [  2.560 s]
>>>>>>>> [INFO] Spark Project Networking ........................... SUCCESS
>>>>>>>> [  6.533 s]
>>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS
>>>>>>>> [  4.176 s]
>>>>>>>> [INFO] Spark Project Unsafe ............................... SUCCESS
>>>>>>>> [  4.809 s]
>>>>>>>> [INFO] Spark Project Launcher ............................. SUCCESS
>>>>>>>> [  6.242 s]
>>>>>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>>>>>> [01:20 min]
>>>>>>>> [INFO] Spark Project GraphX ............................... SUCCESS
>>>>>>>> [  9.148 s]
>>>>>>>> [INFO] Spark Project Streaming ............................ SUCCESS
>>>>>>>> [ 22.760 s]
>>>>>>>> [INFO] Spark Project Catalyst ............................. SUCCESS
>>>>>>>> [ 50.783 s]
>>>>>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>>>>>> [01:05 min]
>>>>>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS
>>>>>>>> [  4.281 s]
>>>>>>>> [INFO] Spark Project ML Library ........................... SUCCESS
>>>>>>>> [ 54.537 s]
>>>>>>>> [INFO] Spark Project Tools ................................ SUCCESS
>>>>>>>> [  0.747 s]
>>>>>>>> [INFO] Spark Project Hive ................................. SUCCESS
>>>>>>>> [ 33.032 s]
>>>>>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS
>>>>>>>> [  3.198 s]
>>>>>>>> [INFO] Spark Project REPL ................................. SUCCESS
>>>>>>>> [  3.573 s]
>>>>>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
>>>>>>>> [  4.617 s]
>>>>>>>> [INFO] Spark Project YARN ................................. SUCCESS
>>>>>>>> [  7.321 s]
>>>>>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS
>>>>>>>> [ 16.496 s]
>>>>>>>> [INFO] Spark Project Assembly ............................. SUCCESS
>>>>>>>> [  2.300 s]
>>>>>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS
>>>>>>>> [  4.219 s]
>>>>>>>> [INFO] Spark Project External Flume ....................... SUCCESS
>>>>>>>> [  6.987 s]
>>>>>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS
>>>>>>>> [  1.465 s]
>>>>>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS
>>>>>>>> [  6.891 s]
>>>>>>>> [INFO] Spark Project Examples ............................. SUCCESS
>>>>>>>> [ 13.465 s]
>>>>>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS
>>>>>>>> [  2.815 s]
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD SUCCESS
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 07:04 min
>>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>>
>>>>>>>> I think it's a good idea. Although releases have been preceded
>>>>>>>> before
>>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>>> formal
>>>>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>>>>> major release. Better to have a little more testing in the wild to
>>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>>
>>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> In the past the Apache Spark community have created preview
>>>>>>>> packages (not
>>>>>>>> official releases) and used those as opportunities to ask community
>>>>>>>> members
>>>>>>>> to test the upcoming versions of Apache Spark. Several people in
>>>>>>>> the Apache
>>>>>>>> community have suggested we conduct votes for these preview
>>>>>>>> packages and
>>>>>>>> turn them into formal releases by the Apache foundation's standard.
>>>>>>>> Preview
>>>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>>>> likely
>>>>>>>> will contain critical bugs or documentation errors, but we will be
>>>>>>>> able to
>>>>>>>> post them to the project's website to get wider feedback. They
>>>>>>>> should
>>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>>> licenses.
>>>>>>>>
>>>>>>>>
>>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>>> version
>>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00
>>>>>>>> PM PDT
>>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>>
>>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>>
>>>>>>>> To learn more about Apache Spark, please see
>>>>>>>> http://spark.apache.org/
>>>>>>>>
>>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>>
>>>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>>>> at:
>>>>>>>>
>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>>
>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>>
>>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>>
>>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>>
>>>>>>>> The list of resolved issues are:
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>>
>>>>>>>>
>>>>>>>> If you are a Spark user, you can help us test this release by
>>>>>>>> taking an
>>>>>>>> existing Apache Spark workload and running on this candidate, then
>>>>>>>> reporting
>>>>>>>> any regressions.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Reynold Xin <rx...@databricks.com>.
Xiao thanks for posting. Please file a bug in JIRA. Again as I said in the
email this is not meant to be a functional release and will contain bugs.

On Thu, May 19, 2016 at 1:20 PM, Xiao Li <ga...@gmail.com> wrote:

> -1
>
> Unable to use Hive meta-store in pyspark shell. Tried both HiveContext and
> SparkSession. Both failed. It always uses in-memory catalog. Anybody else
> hit the same issue?
>
>
> Method 1: SparkSession
>
> >>> from pyspark.sql import SparkSession
>
> >>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
>
> >>>
>
> >>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>
> DataFrame[]
>
> >>> spark.sql("LOAD DATA LOCAL INPATH
> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>
> Traceback (most recent call last):
>
>   File "<stdin>", line 1, in <module>
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
> line 494, in sql
>
>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
> line 933, in __call__
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
> line 57, in deco
>
>     return f(*a, **kw)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
> line 312, in get_return_value
>
> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>
> : java.lang.UnsupportedOperationException: loadTable is not implemented
>
> at
> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>
> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>
> at py4j.Gateway.invoke(Gateway.java:280)
>
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>
> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Method 2: Using HiveContext:
>
> >>> from pyspark.sql import HiveContext
>
> >>> sqlContext = HiveContext(sc)
>
> >>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value
> STRING)")
>
> DataFrame[]
>
> >>> sqlContext.sql("LOAD DATA LOCAL INPATH
> 'examples/src/main/resources/kv1.txt' INTO TABLE src")
>
> Traceback (most recent call last):
>
>   File "<stdin>", line 1, in <module>
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
> line 346, in sql
>
>     return self.sparkSession.sql(sqlQuery)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
> line 494, in sql
>
>     return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
> line 933, in __call__
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
> line 57, in deco
>
>     return f(*a, **kw)
>
>   File
> "/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
> line 312, in get_return_value
>
> py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.
>
> : java.lang.UnsupportedOperationException: loadTable is not implemented
>
> at
> org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)
>
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)
>
> at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)
>
> at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)
>
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
>
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>
> at py4j.Gateway.invoke(Gateway.java:280)
>
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
>
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>
> at py4j.GatewayConnection.run(GatewayConnection.java:211)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
> hvanhovell@questtec.nl>:
>
>> +1
>>
>>
>> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>>
>>> +1
>>>
>>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>>> wrote:
>>>>
>>>>> Hi Ovidiu-Cristian ,
>>>>>
>>>>> The best source of truth is change the filter with target version to
>>>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>
>>>>>> Yes, I can filter..
>>>>>> Did that and for example:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>>
>>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>>> priority and will released maybe with 2.1 for example?
>>>>>>
>>>>>> Keep up the good work!
>>>>>>
>>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>>
>>>>>> You can find that by changing the filter to target version = 2.0.0.
>>>>>> Cheers.
>>>>>>
>>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>>
>>>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>>>> known issue you plan to stay with this release?
>>>>>>>
>>>>>>> with
>>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>>
>>>>>>> mvn -version
>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>>> Java home:
>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family:
>>>>>>> “mac"
>>>>>>>
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] Spark Project Parent POM ........................... SUCCESS
>>>>>>> [  2.635 s]
>>>>>>> [INFO] Spark Project Tags ................................. SUCCESS
>>>>>>> [  1.896 s]
>>>>>>> [INFO] Spark Project Sketch ............................... SUCCESS
>>>>>>> [  2.560 s]
>>>>>>> [INFO] Spark Project Networking ........................... SUCCESS
>>>>>>> [  6.533 s]
>>>>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS
>>>>>>> [  4.176 s]
>>>>>>> [INFO] Spark Project Unsafe ............................... SUCCESS
>>>>>>> [  4.809 s]
>>>>>>> [INFO] Spark Project Launcher ............................. SUCCESS
>>>>>>> [  6.242 s]
>>>>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>>>>> [01:20 min]
>>>>>>> [INFO] Spark Project GraphX ............................... SUCCESS
>>>>>>> [  9.148 s]
>>>>>>> [INFO] Spark Project Streaming ............................ SUCCESS
>>>>>>> [ 22.760 s]
>>>>>>> [INFO] Spark Project Catalyst ............................. SUCCESS
>>>>>>> [ 50.783 s]
>>>>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>>>>> [01:05 min]
>>>>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS
>>>>>>> [  4.281 s]
>>>>>>> [INFO] Spark Project ML Library ........................... SUCCESS
>>>>>>> [ 54.537 s]
>>>>>>> [INFO] Spark Project Tools ................................ SUCCESS
>>>>>>> [  0.747 s]
>>>>>>> [INFO] Spark Project Hive ................................. SUCCESS
>>>>>>> [ 33.032 s]
>>>>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS
>>>>>>> [  3.198 s]
>>>>>>> [INFO] Spark Project REPL ................................. SUCCESS
>>>>>>> [  3.573 s]
>>>>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
>>>>>>> [  4.617 s]
>>>>>>> [INFO] Spark Project YARN ................................. SUCCESS
>>>>>>> [  7.321 s]
>>>>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS
>>>>>>> [ 16.496 s]
>>>>>>> [INFO] Spark Project Assembly ............................. SUCCESS
>>>>>>> [  2.300 s]
>>>>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS
>>>>>>> [  4.219 s]
>>>>>>> [INFO] Spark Project External Flume ....................... SUCCESS
>>>>>>> [  6.987 s]
>>>>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS
>>>>>>> [  1.465 s]
>>>>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS
>>>>>>> [  6.891 s]
>>>>>>> [INFO] Spark Project Examples ............................. SUCCESS
>>>>>>> [ 13.465 s]
>>>>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS
>>>>>>> [  2.815 s]
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD SUCCESS
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 07:04 min
>>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>>> [INFO] Final Memory: 90M/824M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>
>>>>>>> I think it's a good idea. Although releases have been preceded before
>>>>>>> by release candidates for developers, it would be good to get a
>>>>>>> formal
>>>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>>>> major release. Better to have a little more testing in the wild to
>>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>>
>>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> In the past the Apache Spark community have created preview packages
>>>>>>> (not
>>>>>>> official releases) and used those as opportunities to ask community
>>>>>>> members
>>>>>>> to test the upcoming versions of Apache Spark. Several people in the
>>>>>>> Apache
>>>>>>> community have suggested we conduct votes for these preview packages
>>>>>>> and
>>>>>>> turn them into formal releases by the Apache foundation's standard.
>>>>>>> Preview
>>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>>> likely
>>>>>>> will contain critical bugs or documentation errors, but we will be
>>>>>>> able to
>>>>>>> post them to the project's website to get wider feedback. They should
>>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>>> licenses.
>>>>>>>
>>>>>>>
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>> version
>>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00
>>>>>>> PM PDT
>>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>>
>>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>>> at:
>>>>>>>
>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>>
>>>>>>> Release artifacts are signed with the following key:
>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>
>>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>>
>>>>>>> The list of resolved issues are:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>>
>>>>>>>
>>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>>> an
>>>>>>> existing Apache Spark workload and running on this candidate, then
>>>>>>> reporting
>>>>>>> any regressions.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Xiao Li <ga...@gmail.com>.
-1

Unable to use Hive meta-store in pyspark shell. Tried both HiveContext and
SparkSession. Both failed. It always uses in-memory catalog. Anybody else
hit the same issue?


Method 1: SparkSession

>>> from pyspark.sql import SparkSession

>>> spark = SparkSession.builder.enableHiveSupport().getOrCreate()

>>>

>>> spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")

DataFrame[]

>>> spark.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt'
INTO TABLE src")

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
line 494, in sql

    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
line 933, in __call__

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
line 57, in deco

    return f(*a, **kw)

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
line 312, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.

: java.lang.UnsupportedOperationException: loadTable is not implemented

at
org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)

at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)

at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)

at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)

at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)

at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)

at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)

at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)

at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)

at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)

at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:280)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:211)

at java.lang.Thread.run(Thread.java:745)


Method 2: Using HiveContext:

>>> from pyspark.sql import HiveContext

>>> sqlContext = HiveContext(sc)

>>> sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")

DataFrame[]

>>> sqlContext.sql("LOAD DATA LOCAL INPATH
'examples/src/main/resources/kv1.txt' INTO TABLE src")

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/context.py",
line 346, in sql

    return self.sparkSession.sql(sqlQuery)

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/session.py",
line 494, in sql

    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
line 933, in __call__

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/pyspark/sql/utils.py",
line 57, in deco

    return f(*a, **kw)

  File
"/Users/xiaoli/IdeaProjects/sparkDelivery/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py",
line 312, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o21.sql.

: java.lang.UnsupportedOperationException: loadTable is not implemented

at
org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.loadTable(InMemoryCatalog.scala:297)

at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:280)

at org.apache.spark.sql.execution.command.LoadData.run(tables.scala:263)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)

at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)

at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)

at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)

at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)

at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:85)

at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:85)

at org.apache.spark.sql.Dataset.<init>(Dataset.scala:187)

at org.apache.spark.sql.Dataset.<init>(Dataset.scala:168)

at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)

at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:280)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:211)

at java.lang.Thread.run(Thread.java:745)

2016-05-19 12:49 GMT-07:00 Herman van Hövell tot Westerflier <
hvanhovell@questtec.nl>:

> +1
>
>
> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>
>> +1
>>
>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>> wrote:
>>
>>> +1
>>>
>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>> wrote:
>>>
>>>> Hi Ovidiu-Cristian ,
>>>>
>>>> The best source of truth is change the filter with target version to
>>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>
>>>>
>>>>
>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>
>>>>> Yes, I can filter..
>>>>> Did that and for example:
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>
>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>> priority and will released maybe with 2.1 for example?
>>>>>
>>>>> Keep up the good work!
>>>>>
>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>
>>>>> You can find that by changing the filter to target version = 2.0.0.
>>>>> Cheers.
>>>>>
>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>
>>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>>> known issue you plan to stay with this release?
>>>>>>
>>>>>> with
>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>
>>>>>> mvn -version
>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>> Java home:
>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>>>>>
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] Spark Project Parent POM ........................... SUCCESS
>>>>>> [  2.635 s]
>>>>>> [INFO] Spark Project Tags ................................. SUCCESS
>>>>>> [  1.896 s]
>>>>>> [INFO] Spark Project Sketch ............................... SUCCESS
>>>>>> [  2.560 s]
>>>>>> [INFO] Spark Project Networking ........................... SUCCESS
>>>>>> [  6.533 s]
>>>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS
>>>>>> [  4.176 s]
>>>>>> [INFO] Spark Project Unsafe ............................... SUCCESS
>>>>>> [  4.809 s]
>>>>>> [INFO] Spark Project Launcher ............................. SUCCESS
>>>>>> [  6.242 s]
>>>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>>>> [01:20 min]
>>>>>> [INFO] Spark Project GraphX ............................... SUCCESS
>>>>>> [  9.148 s]
>>>>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>>>>> 22.760 s]
>>>>>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>>>>>> 50.783 s]
>>>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>>>> [01:05 min]
>>>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS
>>>>>> [  4.281 s]
>>>>>> [INFO] Spark Project ML Library ........................... SUCCESS [
>>>>>> 54.537 s]
>>>>>> [INFO] Spark Project Tools ................................ SUCCESS
>>>>>> [  0.747 s]
>>>>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>>>>> 33.032 s]
>>>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS
>>>>>> [  3.198 s]
>>>>>> [INFO] Spark Project REPL ................................. SUCCESS
>>>>>> [  3.573 s]
>>>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
>>>>>> [  4.617 s]
>>>>>> [INFO] Spark Project YARN ................................. SUCCESS
>>>>>> [  7.321 s]
>>>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>>>>>> 16.496 s]
>>>>>> [INFO] Spark Project Assembly ............................. SUCCESS
>>>>>> [  2.300 s]
>>>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS
>>>>>> [  4.219 s]
>>>>>> [INFO] Spark Project External Flume ....................... SUCCESS
>>>>>> [  6.987 s]
>>>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS
>>>>>> [  1.465 s]
>>>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS
>>>>>> [  6.891 s]
>>>>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>>>>> 13.465 s]
>>>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS
>>>>>> [  2.815 s]
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] BUILD SUCCESS
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Total time: 07:04 min
>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>> [INFO] Final Memory: 90M/824M
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>
>>>>>> I think it's a good idea. Although releases have been preceded before
>>>>>> by release candidates for developers, it would be good to get a formal
>>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>>> major release. Better to have a little more testing in the wild to
>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>
>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>
>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In the past the Apache Spark community have created preview packages
>>>>>> (not
>>>>>> official releases) and used those as opportunities to ask community
>>>>>> members
>>>>>> to test the upcoming versions of Apache Spark. Several people in the
>>>>>> Apache
>>>>>> community have suggested we conduct votes for these preview packages
>>>>>> and
>>>>>> turn them into formal releases by the Apache foundation's standard.
>>>>>> Preview
>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>> likely
>>>>>> will contain critical bugs or documentation errors, but we will be
>>>>>> able to
>>>>>> post them to the project's website to get wider feedback. They should
>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>> licenses.
>>>>>>
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>> version
>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00
>>>>>> PM PDT
>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>> at:
>>>>>>
>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>
>>>>>> Release artifacts are signed with the following key:
>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>>
>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>
>>>>>> The list of resolved issues are:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an
>>>>>> existing Apache Spark workload and running on this candidate, then
>>>>>> reporting
>>>>>> any regressions.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by vishnu prasad <vi...@gmail.com>.
+1

On 20 May 2016 at 01:19, Herman van Hövell tot Westerflier <
hvanhovell@questtec.nl> wrote:

> +1
>
>
> 2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:
>
>> +1
>>
>> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
>> wrote:
>>
>>> +1
>>>
>>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>>> wrote:
>>>
>>>> Hi Ovidiu-Cristian ,
>>>>
>>>> The best source of truth is change the filter with target version to
>>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>>
>>>>
>>>>
>>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>
>>>>> Yes, I can filter..
>>>>> Did that and for example:
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>>
>>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>>> priority and will released maybe with 2.1 for example?
>>>>>
>>>>> Keep up the good work!
>>>>>
>>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>>
>>>>> You can find that by changing the filter to target version = 2.0.0.
>>>>> Cheers.
>>>>>
>>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>>
>>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>>> known issue you plan to stay with this release?
>>>>>>
>>>>>> with
>>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>>
>>>>>> mvn -version
>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>>> 2015-11-10T17:41:47+01:00)
>>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>>> Java home:
>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>>>>>
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] Spark Project Parent POM ........................... SUCCESS
>>>>>> [  2.635 s]
>>>>>> [INFO] Spark Project Tags ................................. SUCCESS
>>>>>> [  1.896 s]
>>>>>> [INFO] Spark Project Sketch ............................... SUCCESS
>>>>>> [  2.560 s]
>>>>>> [INFO] Spark Project Networking ........................... SUCCESS
>>>>>> [  6.533 s]
>>>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS
>>>>>> [  4.176 s]
>>>>>> [INFO] Spark Project Unsafe ............................... SUCCESS
>>>>>> [  4.809 s]
>>>>>> [INFO] Spark Project Launcher ............................. SUCCESS
>>>>>> [  6.242 s]
>>>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>>>> [01:20 min]
>>>>>> [INFO] Spark Project GraphX ............................... SUCCESS
>>>>>> [  9.148 s]
>>>>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>>>>> 22.760 s]
>>>>>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>>>>>> 50.783 s]
>>>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>>>> [01:05 min]
>>>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS
>>>>>> [  4.281 s]
>>>>>> [INFO] Spark Project ML Library ........................... SUCCESS [
>>>>>> 54.537 s]
>>>>>> [INFO] Spark Project Tools ................................ SUCCESS
>>>>>> [  0.747 s]
>>>>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>>>>> 33.032 s]
>>>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS
>>>>>> [  3.198 s]
>>>>>> [INFO] Spark Project REPL ................................. SUCCESS
>>>>>> [  3.573 s]
>>>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS
>>>>>> [  4.617 s]
>>>>>> [INFO] Spark Project YARN ................................. SUCCESS
>>>>>> [  7.321 s]
>>>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>>>>>> 16.496 s]
>>>>>> [INFO] Spark Project Assembly ............................. SUCCESS
>>>>>> [  2.300 s]
>>>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS
>>>>>> [  4.219 s]
>>>>>> [INFO] Spark Project External Flume ....................... SUCCESS
>>>>>> [  6.987 s]
>>>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS
>>>>>> [  1.465 s]
>>>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS
>>>>>> [  6.891 s]
>>>>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>>>>> 13.465 s]
>>>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS
>>>>>> [  2.815 s]
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] BUILD SUCCESS
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Total time: 07:04 min
>>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>>> [INFO] Final Memory: 90M/824M
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>>
>>>>>> I think it's a good idea. Although releases have been preceded before
>>>>>> by release candidates for developers, it would be good to get a formal
>>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>>> major release. Better to have a little more testing in the wild to
>>>>>> identify problems before 2.0.0 is finalized.
>>>>>>
>>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>>
>>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In the past the Apache Spark community have created preview packages
>>>>>> (not
>>>>>> official releases) and used those as opportunities to ask community
>>>>>> members
>>>>>> to test the upcoming versions of Apache Spark. Several people in the
>>>>>> Apache
>>>>>> community have suggested we conduct votes for these preview packages
>>>>>> and
>>>>>> turn them into formal releases by the Apache foundation's standard.
>>>>>> Preview
>>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>>> likely
>>>>>> will contain critical bugs or documentation errors, but we will be
>>>>>> able to
>>>>>> post them to the project's website to get wider feedback. They should
>>>>>> satisfy the legal requirements of Apache's release policy
>>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>>> licenses.
>>>>>>
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>> version
>>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00
>>>>>> PM PDT
>>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is 2.0.0-preview
>>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>> at:
>>>>>>
>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>>
>>>>>> Release artifacts are signed with the following key:
>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>>
>>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>>
>>>>>> The list of resolved issues are:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>>
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an
>>>>>> existing Apache Spark workload and running on this candidate, then
>>>>>> reporting
>>>>>> any regressions.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Herman van Hövell tot Westerflier <hv...@questtec.nl>.
+1


2016-05-19 18:20 GMT+02:00 Xiangrui Meng <me...@databricks.com>:

> +1
>
> On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
> wrote:
>
>> +1
>>
>> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com>
>> wrote:
>>
>>> Hi Ovidiu-Cristian ,
>>>
>>> The best source of truth is change the filter with target version to
>>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>>
>>>
>>>
>>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>
>>>> Yes, I can filter..
>>>> Did that and for example:
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>>
>>>> To rephrase: for 2.0 do you have specific issues that are not a
>>>> priority and will released maybe with 2.1 for example?
>>>>
>>>> Keep up the good work!
>>>>
>>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>>
>>>> You can find that by changing the filter to target version = 2.0.0.
>>>> Cheers.
>>>>
>>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>>
>>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>>> known issue you plan to stay with this release?
>>>>>
>>>>> with
>>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>>> -Phive-thriftserver -DskipTests clean package
>>>>>
>>>>> mvn -version
>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>>> 2015-11-10T17:41:47+01:00)
>>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>>> Java home:
>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>>>>
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>>>>> 2.635 s]
>>>>> [INFO] Spark Project Tags ................................. SUCCESS [
>>>>> 1.896 s]
>>>>> [INFO] Spark Project Sketch ............................... SUCCESS [
>>>>> 2.560 s]
>>>>> [INFO] Spark Project Networking ........................... SUCCESS [
>>>>> 6.533 s]
>>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>>>>> 4.176 s]
>>>>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>>>>> 4.809 s]
>>>>> [INFO] Spark Project Launcher ............................. SUCCESS [
>>>>> 6.242 s]
>>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>>> [01:20 min]
>>>>> [INFO] Spark Project GraphX ............................... SUCCESS [
>>>>> 9.148 s]
>>>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>>>> 22.760 s]
>>>>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>>>>> 50.783 s]
>>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>>> [01:05 min]
>>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>>>>> 4.281 s]
>>>>> [INFO] Spark Project ML Library ........................... SUCCESS [
>>>>> 54.537 s]
>>>>> [INFO] Spark Project Tools ................................ SUCCESS [
>>>>> 0.747 s]
>>>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>>>> 33.032 s]
>>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [
>>>>> 3.198 s]
>>>>> [INFO] Spark Project REPL ................................. SUCCESS [
>>>>> 3.573 s]
>>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [
>>>>> 4.617 s]
>>>>> [INFO] Spark Project YARN ................................. SUCCESS [
>>>>> 7.321 s]
>>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>>>>> 16.496 s]
>>>>> [INFO] Spark Project Assembly ............................. SUCCESS [
>>>>> 2.300 s]
>>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS [
>>>>> 4.219 s]
>>>>> [INFO] Spark Project External Flume ....................... SUCCESS [
>>>>> 6.987 s]
>>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS [
>>>>> 1.465 s]
>>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [
>>>>> 6.891 s]
>>>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>>>> 13.465 s]
>>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [
>>>>> 2.815 s]
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD SUCCESS
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 07:04 min
>>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>>> [INFO] Final Memory: 90M/824M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>>
>>>>> I think it's a good idea. Although releases have been preceded before
>>>>> by release candidates for developers, it would be good to get a formal
>>>>> preview/beta release ratified for public consumption ahead of a new
>>>>> major release. Better to have a little more testing in the wild to
>>>>> identify problems before 2.0.0 is finalized.
>>>>>
>>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>>
>>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> In the past the Apache Spark community have created preview packages
>>>>> (not
>>>>> official releases) and used those as opportunities to ask community
>>>>> members
>>>>> to test the upcoming versions of Apache Spark. Several people in the
>>>>> Apache
>>>>> community have suggested we conduct votes for these preview packages
>>>>> and
>>>>> turn them into formal releases by the Apache foundation's standard.
>>>>> Preview
>>>>> releases are not meant to be functional, i.e. they can and highly
>>>>> likely
>>>>> will contain critical bugs or documentation errors, but we will be
>>>>> able to
>>>>> post them to the project's website to get wider feedback. They should
>>>>> satisfy the legal requirements of Apache's release policy
>>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>>> licenses.
>>>>>
>>>>>
>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>> version
>>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
>>>>> PDT
>>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>>
>>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>
>>>>> The tag to be voted on is 2.0.0-preview
>>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>>
>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>
>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>>
>>>>> Release artifacts are signed with the following key:
>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>
>>>>> The documentation corresponding to this release can be found at:
>>>>>
>>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>>
>>>>> The list of resolved issues are:
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>>
>>>>>
>>>>> If you are a Spark user, you can help us test this release by taking an
>>>>> existing Apache Spark workload and running on this candidate, then
>>>>> reporting
>>>>> any regressions.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Xiangrui Meng <me...@databricks.com>.
+1

On Thu, May 19, 2016 at 9:18 AM Joseph Bradley <jo...@databricks.com>
wrote:

> +1
>
> On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Hi Ovidiu-Cristian ,
>>
>> The best source of truth is change the filter with target version to
>> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
>> get closer to 2.0 release, more will be retargeted at 2.1.0.
>>
>>
>>
>> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
>> ovidiu-cristian.marcu@inria.fr> wrote:
>>
>>> Yes, I can filter..
>>> Did that and for example:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>>
>>> To rephrase: for 2.0 do you have specific issues that are not a priority
>>> and will released maybe with 2.1 for example?
>>>
>>> Keep up the good work!
>>>
>>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>>
>>> You can find that by changing the filter to target version = 2.0.0.
>>> Cheers.
>>>
>>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>>> ovidiu-cristian.marcu@inria.fr> wrote:
>>>
>>>> +1 Great, I see the list of resolved issues, do you have a list of
>>>> known issue you plan to stay with this release?
>>>>
>>>> with
>>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>>> -Phive-thriftserver -DskipTests clean package
>>>>
>>>> mvn -version
>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>>> 2015-11-10T17:41:47+01:00)
>>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>>> Java home:
>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>>> Default locale: en_US, platform encoding: UTF-8
>>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>>>
>>>> [INFO] Reactor Summary:
>>>> [INFO]
>>>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>>>> 2.635 s]
>>>> [INFO] Spark Project Tags ................................. SUCCESS [
>>>> 1.896 s]
>>>> [INFO] Spark Project Sketch ............................... SUCCESS [
>>>> 2.560 s]
>>>> [INFO] Spark Project Networking ........................... SUCCESS [
>>>> 6.533 s]
>>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>>>> 4.176 s]
>>>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>>>> 4.809 s]
>>>> [INFO] Spark Project Launcher ............................. SUCCESS [
>>>> 6.242 s]
>>>> [INFO] Spark Project Core ................................. SUCCESS
>>>> [01:20 min]
>>>> [INFO] Spark Project GraphX ............................... SUCCESS [
>>>> 9.148 s]
>>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>>> 22.760 s]
>>>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>>>> 50.783 s]
>>>> [INFO] Spark Project SQL .................................. SUCCESS
>>>> [01:05 min]
>>>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>>>> 4.281 s]
>>>> [INFO] Spark Project ML Library ........................... SUCCESS [
>>>> 54.537 s]
>>>> [INFO] Spark Project Tools ................................ SUCCESS [
>>>> 0.747 s]
>>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>>> 33.032 s]
>>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [
>>>> 3.198 s]
>>>> [INFO] Spark Project REPL ................................. SUCCESS [
>>>> 3.573 s]
>>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [
>>>> 4.617 s]
>>>> [INFO] Spark Project YARN ................................. SUCCESS [
>>>> 7.321 s]
>>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>>>> 16.496 s]
>>>> [INFO] Spark Project Assembly ............................. SUCCESS [
>>>> 2.300 s]
>>>> [INFO] Spark Project External Flume Sink .................. SUCCESS [
>>>> 4.219 s]
>>>> [INFO] Spark Project External Flume ....................... SUCCESS [
>>>> 6.987 s]
>>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS [
>>>> 1.465 s]
>>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [
>>>> 6.891 s]
>>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>>> 13.465 s]
>>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [
>>>> 2.815 s]
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD SUCCESS
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 07:04 min
>>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>>> [INFO] Final Memory: 90M/824M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>>
>>>> I think it's a good idea. Although releases have been preceded before
>>>> by release candidates for developers, it would be good to get a formal
>>>> preview/beta release ratified for public consumption ahead of a new
>>>> major release. Better to have a little more testing in the wild to
>>>> identify problems before 2.0.0 is finalized.
>>>>
>>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>>> -Phive-thriftserver -Phadoop-2.6".
>>>>
>>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> In the past the Apache Spark community have created preview packages
>>>> (not
>>>> official releases) and used those as opportunities to ask community
>>>> members
>>>> to test the upcoming versions of Apache Spark. Several people in the
>>>> Apache
>>>> community have suggested we conduct votes for these preview packages and
>>>> turn them into formal releases by the Apache foundation's standard.
>>>> Preview
>>>> releases are not meant to be functional, i.e. they can and highly likely
>>>> will contain critical bugs or documentation errors, but we will be able
>>>> to
>>>> post them to the project's website to get wider feedback. They should
>>>> satisfy the legal requirements of Apache's release policy
>>>> (http://www.apache.org/dev/release.html) such as having proper
>>>> licenses.
>>>>
>>>>
>>>> Please vote on releasing the following candidate as Apache Spark version
>>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
>>>> PDT
>>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>>
>>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>>> [ ] -1 Do not release this package because ...
>>>>
>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>
>>>> The tag to be voted on is 2.0.0-preview
>>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>>
>>>> The release files, including signatures, digests, etc. can be found at:
>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>>
>>>> Release artifacts are signed with the following key:
>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>
>>>> The documentation corresponding to this release can be found at:
>>>>
>>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>>
>>>> The list of resolved issues are:
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>>
>>>>
>>>> If you are a Spark user, you can help us test this release by taking an
>>>> existing Apache Spark workload and running on this candidate, then
>>>> reporting
>>>> any regressions.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Joseph Bradley <jo...@databricks.com>.
+1

On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com> wrote:

> Hi Ovidiu-Cristian ,
>
> The best source of truth is change the filter with target version to
> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
> get closer to 2.0 release, more will be retargeted at 2.1.0.
>
>
>
> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
> ovidiu-cristian.marcu@inria.fr> wrote:
>
>> Yes, I can filter..
>> Did that and for example:
>>
>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>
>> To rephrase: for 2.0 do you have specific issues that are not a priority
>> and will released maybe with 2.1 for example?
>>
>> Keep up the good work!
>>
>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>
>> You can find that by changing the filter to target version = 2.0.0.
>> Cheers.
>>
>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>> ovidiu-cristian.marcu@inria.fr> wrote:
>>
>>> +1 Great, I see the list of resolved issues, do you have a list of known
>>> issue you plan to stay with this release?
>>>
>>> with
>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>> -Phive-thriftserver -DskipTests clean package
>>>
>>> mvn -version
>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>> 2015-11-10T17:41:47+01:00)
>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>> Java home:
>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>>
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>>> 2.635 s]
>>> [INFO] Spark Project Tags ................................. SUCCESS [
>>> 1.896 s]
>>> [INFO] Spark Project Sketch ............................... SUCCESS [
>>> 2.560 s]
>>> [INFO] Spark Project Networking ........................... SUCCESS [
>>> 6.533 s]
>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>>> 4.176 s]
>>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>>> 4.809 s]
>>> [INFO] Spark Project Launcher ............................. SUCCESS [
>>> 6.242 s]
>>> [INFO] Spark Project Core ................................. SUCCESS
>>> [01:20 min]
>>> [INFO] Spark Project GraphX ............................... SUCCESS [
>>> 9.148 s]
>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>> 22.760 s]
>>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>>> 50.783 s]
>>> [INFO] Spark Project SQL .................................. SUCCESS
>>> [01:05 min]
>>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>>> 4.281 s]
>>> [INFO] Spark Project ML Library ........................... SUCCESS [
>>> 54.537 s]
>>> [INFO] Spark Project Tools ................................ SUCCESS [
>>> 0.747 s]
>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>> 33.032 s]
>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [
>>> 3.198 s]
>>> [INFO] Spark Project REPL ................................. SUCCESS [
>>> 3.573 s]
>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [
>>> 4.617 s]
>>> [INFO] Spark Project YARN ................................. SUCCESS [
>>> 7.321 s]
>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>>> 16.496 s]
>>> [INFO] Spark Project Assembly ............................. SUCCESS [
>>> 2.300 s]
>>> [INFO] Spark Project External Flume Sink .................. SUCCESS [
>>> 4.219 s]
>>> [INFO] Spark Project External Flume ....................... SUCCESS [
>>> 6.987 s]
>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS [
>>> 1.465 s]
>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [
>>> 6.891 s]
>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>> 13.465 s]
>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [
>>> 2.815 s]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 07:04 min
>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>> [INFO] Final Memory: 90M/824M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> I think it's a good idea. Although releases have been preceded before
>>> by release candidates for developers, it would be good to get a formal
>>> preview/beta release ratified for public consumption ahead of a new
>>> major release. Better to have a little more testing in the wild to
>>> identify problems before 2.0.0 is finalized.
>>>
>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>> -Phive-thriftserver -Phadoop-2.6".
>>>
>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>>
>>> Hi,
>>>
>>> In the past the Apache Spark community have created preview packages (not
>>> official releases) and used those as opportunities to ask community
>>> members
>>> to test the upcoming versions of Apache Spark. Several people in the
>>> Apache
>>> community have suggested we conduct votes for these preview packages and
>>> turn them into formal releases by the Apache foundation's standard.
>>> Preview
>>> releases are not meant to be functional, i.e. they can and highly likely
>>> will contain critical bugs or documentation errors, but we will be able
>>> to
>>> post them to the project's website to get wider feedback. They should
>>> satisfy the legal requirements of Apache's release policy
>>> (http://www.apache.org/dev/release.html) such as having proper licenses.
>>>
>>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
>>> PDT
>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is 2.0.0-preview
>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>
>>> The list of resolved issues are:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>
>>>
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Apache Spark workload and running on this candidate, then
>>> reporting
>>> any regressions.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>>
>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Yin Huai <yh...@databricks.com>.
+1

On Wed, May 18, 2016 at 10:49 AM, Reynold Xin <rx...@databricks.com> wrote:

> Hi Ovidiu-Cristian ,
>
> The best source of truth is change the filter with target version to
> 2.1.0. Not a lot of tickets have been targeted yet, but I'd imagine as we
> get closer to 2.0 release, more will be retargeted at 2.1.0.
>
>
>
> On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
> ovidiu-cristian.marcu@inria.fr> wrote:
>
>> Yes, I can filter..
>> Did that and for example:
>>
>> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
>> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>>
>> To rephrase: for 2.0 do you have specific issues that are not a priority
>> and will released maybe with 2.1 for example?
>>
>> Keep up the good work!
>>
>> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>>
>> You can find that by changing the filter to target version = 2.0.0.
>> Cheers.
>>
>> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
>> ovidiu-cristian.marcu@inria.fr> wrote:
>>
>>> +1 Great, I see the list of resolved issues, do you have a list of known
>>> issue you plan to stay with this release?
>>>
>>> with
>>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>>> -Phive-thriftserver -DskipTests clean package
>>>
>>> mvn -version
>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>> 2015-11-10T17:41:47+01:00)
>>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>>> Java version: 1.7.0_80, vendor: Oracle Corporation
>>> Java home:
>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>>
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>>> 2.635 s]
>>> [INFO] Spark Project Tags ................................. SUCCESS [
>>> 1.896 s]
>>> [INFO] Spark Project Sketch ............................... SUCCESS [
>>> 2.560 s]
>>> [INFO] Spark Project Networking ........................... SUCCESS [
>>> 6.533 s]
>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>>> 4.176 s]
>>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>>> 4.809 s]
>>> [INFO] Spark Project Launcher ............................. SUCCESS [
>>> 6.242 s]
>>> [INFO] Spark Project Core ................................. SUCCESS
>>> [01:20 min]
>>> [INFO] Spark Project GraphX ............................... SUCCESS [
>>> 9.148 s]
>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>> 22.760 s]
>>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>>> 50.783 s]
>>> [INFO] Spark Project SQL .................................. SUCCESS
>>> [01:05 min]
>>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>>> 4.281 s]
>>> [INFO] Spark Project ML Library ........................... SUCCESS [
>>> 54.537 s]
>>> [INFO] Spark Project Tools ................................ SUCCESS [
>>> 0.747 s]
>>> [INFO] Spark Project Hive ................................. SUCCESS [
>>> 33.032 s]
>>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [
>>> 3.198 s]
>>> [INFO] Spark Project REPL ................................. SUCCESS [
>>> 3.573 s]
>>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [
>>> 4.617 s]
>>> [INFO] Spark Project YARN ................................. SUCCESS [
>>> 7.321 s]
>>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>>> 16.496 s]
>>> [INFO] Spark Project Assembly ............................. SUCCESS [
>>> 2.300 s]
>>> [INFO] Spark Project External Flume Sink .................. SUCCESS [
>>> 4.219 s]
>>> [INFO] Spark Project External Flume ....................... SUCCESS [
>>> 6.987 s]
>>> [INFO] Spark Project External Flume Assembly .............. SUCCESS [
>>> 1.465 s]
>>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [
>>> 6.891 s]
>>> [INFO] Spark Project Examples ............................. SUCCESS [
>>> 13.465 s]
>>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [
>>> 2.815 s]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 07:04 min
>>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>>> [INFO] Final Memory: 90M/824M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> I think it's a good idea. Although releases have been preceded before
>>> by release candidates for developers, it would be good to get a formal
>>> preview/beta release ratified for public consumption ahead of a new
>>> major release. Better to have a little more testing in the wild to
>>> identify problems before 2.0.0 is finalized.
>>>
>>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>>> 8, compilation and tests succeed for "-Pyarn -Phive
>>> -Phive-thriftserver -Phadoop-2.6".
>>>
>>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>>
>>> Hi,
>>>
>>> In the past the Apache Spark community have created preview packages (not
>>> official releases) and used those as opportunities to ask community
>>> members
>>> to test the upcoming versions of Apache Spark. Several people in the
>>> Apache
>>> community have suggested we conduct votes for these preview packages and
>>> turn them into formal releases by the Apache foundation's standard.
>>> Preview
>>> releases are not meant to be functional, i.e. they can and highly likely
>>> will contain critical bugs or documentation errors, but we will be able
>>> to
>>> post them to the project's website to get wider feedback. They should
>>> satisfy the legal requirements of Apache's release policy
>>> (http://www.apache.org/dev/release.html) such as having proper licenses.
>>>
>>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
>>> PDT
>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is 2.0.0-preview
>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>
>>> The list of resolved issues are:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>
>>>
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Apache Spark workload and running on this candidate, then
>>> reporting
>>> any regressions.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>>
>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Reynold Xin <rx...@databricks.com>.
Hi Ovidiu-Cristian ,

The best source of truth is change the filter with target version to 2.1.0.
Not a lot of tickets have been targeted yet, but I'd imagine as we get
closer to 2.0 release, more will be retargeted at 2.1.0.



On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.marcu@inria.fr> wrote:

> Yes, I can filter..
> Did that and for example:
>
> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
> <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>
>
> To rephrase: for 2.0 do you have specific issues that are not a priority
> and will released maybe with 2.1 for example?
>
> Keep up the good work!
>
> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
>
> You can find that by changing the filter to target version = 2.0.0. Cheers.
>
> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
> ovidiu-cristian.marcu@inria.fr> wrote:
>
>> +1 Great, I see the list of resolved issues, do you have a list of known
>> issue you plan to stay with this release?
>>
>> with
>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>> -Phive-thriftserver -DskipTests clean package
>>
>> mvn -version
>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>> 2015-11-10T17:41:47+01:00)
>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>> Java version: 1.7.0_80, vendor: Oracle Corporation
>> Java home:
>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>> 2.635 s]
>> [INFO] Spark Project Tags ................................. SUCCESS [
>> 1.896 s]
>> [INFO] Spark Project Sketch ............................... SUCCESS [
>> 2.560 s]
>> [INFO] Spark Project Networking ........................... SUCCESS [
>> 6.533 s]
>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>> 4.176 s]
>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>> 4.809 s]
>> [INFO] Spark Project Launcher ............................. SUCCESS [
>> 6.242 s]
>> [INFO] Spark Project Core ................................. SUCCESS
>> [01:20 min]
>> [INFO] Spark Project GraphX ............................... SUCCESS [
>> 9.148 s]
>> [INFO] Spark Project Streaming ............................ SUCCESS [
>> 22.760 s]
>> [INFO] Spark Project Catalyst ............................. SUCCESS [
>> 50.783 s]
>> [INFO] Spark Project SQL .................................. SUCCESS
>> [01:05 min]
>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>> 4.281 s]
>> [INFO] Spark Project ML Library ........................... SUCCESS [
>> 54.537 s]
>> [INFO] Spark Project Tools ................................ SUCCESS [
>> 0.747 s]
>> [INFO] Spark Project Hive ................................. SUCCESS [
>> 33.032 s]
>> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [
>> 3.198 s]
>> [INFO] Spark Project REPL ................................. SUCCESS [
>> 3.573 s]
>> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [
>> 4.617 s]
>> [INFO] Spark Project YARN ................................. SUCCESS [
>> 7.321 s]
>> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
>> 16.496 s]
>> [INFO] Spark Project Assembly ............................. SUCCESS [
>> 2.300 s]
>> [INFO] Spark Project External Flume Sink .................. SUCCESS [
>> 4.219 s]
>> [INFO] Spark Project External Flume ....................... SUCCESS [
>> 6.987 s]
>> [INFO] Spark Project External Flume Assembly .............. SUCCESS [
>> 1.465 s]
>> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [
>> 6.891 s]
>> [INFO] Spark Project Examples ............................. SUCCESS [
>> 13.465 s]
>> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [
>> 2.815 s]
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 07:04 min
>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>> [INFO] Final Memory: 90M/824M
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>>
>> I think it's a good idea. Although releases have been preceded before
>> by release candidates for developers, it would be good to get a formal
>> preview/beta release ratified for public consumption ahead of a new
>> major release. Better to have a little more testing in the wild to
>> identify problems before 2.0.0 is finalized.
>>
>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>> 8, compilation and tests succeed for "-Pyarn -Phive
>> -Phive-thriftserver -Phadoop-2.6".
>>
>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>>
>> Hi,
>>
>> In the past the Apache Spark community have created preview packages (not
>> official releases) and used those as opportunities to ask community
>> members
>> to test the upcoming versions of Apache Spark. Several people in the
>> Apache
>> community have suggested we conduct votes for these preview packages and
>> turn them into formal releases by the Apache foundation's standard.
>> Preview
>> releases are not meant to be functional, i.e. they can and highly likely
>> will contain critical bugs or documentation errors, but we will be able to
>> post them to the project's website to get wider feedback. They should
>> satisfy the legal requirements of Apache's release policy
>> (http://www.apache.org/dev/release.html) such as having proper licenses.
>>
>>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is 2.0.0-preview
>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The documentation corresponding to this release can be found at:
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>
>> The list of resolved issues are:
>>
>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>
>>
>> If you are a Spark user, you can help us test this release by taking an
>> existing Apache Spark workload and running on this candidate, then
>> reporting
>> any regressions.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>>
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Ovidiu-Cristian MARCU <ov...@inria.fr>.
Yes, I can filter..
Did that and for example:
https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0 <https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20=%20SPARK%20AND%20resolution%20=%20Unresolved%20AND%20affectedVersion%20=%202.0.0>

To rephrase: for 2.0 do you have specific issues that are not a priority and will released maybe with 2.1 for example?

Keep up the good work!

> On 18 May 2016, at 18:19, Reynold Xin <rx...@databricks.com> wrote:
> 
> You can find that by changing the filter to target version = 2.0.0. Cheers.
> 
> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <ovidiu-cristian.marcu@inria.fr <ma...@inria.fr>> wrote:
> +1 Great, I see the list of resolved issues, do you have a list of known issue you plan to stay with this release?
> 
> with
> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive -Phive-thriftserver -DskipTests clean package
> 
> mvn -version
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
> Java version: 1.7.0_80, vendor: Oracle Corporation
> Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Spark Project Parent POM ........................... SUCCESS [  2.635 s]
> [INFO] Spark Project Tags ................................. SUCCESS [  1.896 s]
> [INFO] Spark Project Sketch ............................... SUCCESS [  2.560 s]
> [INFO] Spark Project Networking ........................... SUCCESS [  6.533 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  4.176 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [  4.809 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [  6.242 s]
> [INFO] Spark Project Core ................................. SUCCESS [01:20 min]
> [INFO] Spark Project GraphX ............................... SUCCESS [  9.148 s]
> [INFO] Spark Project Streaming ............................ SUCCESS [ 22.760 s]
> [INFO] Spark Project Catalyst ............................. SUCCESS [ 50.783 s]
> [INFO] Spark Project SQL .................................. SUCCESS [01:05 min]
> [INFO] Spark Project ML Local Library ..................... SUCCESS [  4.281 s]
> [INFO] Spark Project ML Library ........................... SUCCESS [ 54.537 s]
> [INFO] Spark Project Tools ................................ SUCCESS [  0.747 s]
> [INFO] Spark Project Hive ................................. SUCCESS [ 33.032 s]
> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [  3.198 s]
> [INFO] Spark Project REPL ................................. SUCCESS [  3.573 s]
> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [  4.617 s]
> [INFO] Spark Project YARN ................................. SUCCESS [  7.321 s]
> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 16.496 s]
> [INFO] Spark Project Assembly ............................. SUCCESS [  2.300 s]
> [INFO] Spark Project External Flume Sink .................. SUCCESS [  4.219 s]
> [INFO] Spark Project External Flume ....................... SUCCESS [  6.987 s]
> [INFO] Spark Project External Flume Assembly .............. SUCCESS [  1.465 s]
> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [  6.891 s]
> [INFO] Spark Project Examples ............................. SUCCESS [ 13.465 s]
> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [  2.815 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 07:04 min
> [INFO] Finished at: 2016-05-18T17:55:33+02:00
> [INFO] Final Memory: 90M/824M
> [INFO] ------------------------------------------------------------------------
> 
>> On 18 May 2016, at 16:28, Sean Owen <sowen@cloudera.com <ma...@cloudera.com>> wrote:
>> 
>> I think it's a good idea. Although releases have been preceded before
>> by release candidates for developers, it would be good to get a formal
>> preview/beta release ratified for public consumption ahead of a new
>> major release. Better to have a little more testing in the wild to
>> identify problems before 2.0.0 is finalized.
>> 
>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>> 8, compilation and tests succeed for "-Pyarn -Phive
>> -Phive-thriftserver -Phadoop-2.6".
>> 
>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rxin@apache.org <ma...@apache.org>> wrote:
>>> Hi,
>>> 
>>> In the past the Apache Spark community have created preview packages (not
>>> official releases) and used those as opportunities to ask community members
>>> to test the upcoming versions of Apache Spark. Several people in the Apache
>>> community have suggested we conduct votes for these preview packages and
>>> turn them into formal releases by the Apache foundation's standard. Preview
>>> releases are not meant to be functional, i.e. they can and highly likely
>>> will contain critical bugs or documentation errors, but we will be able to
>>> post them to the project's website to get wider feedback. They should
>>> satisfy the legal requirements of Apache's release policy
>>> (http://www.apache.org/dev/release.html <http://www.apache.org/dev/release.html>) such as having proper licenses.
>>> 
>>> 
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>> 
>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>> [ ] -1 Do not release this package because ...
>>> 
>>> To learn more about Apache Spark, please see http://spark.apache.org/ <http://spark.apache.org/>
>>> 
>>> The tag to be voted on is 2.0.0-preview
>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>> 
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/ <http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/>
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc <https://people.apache.org/keys/committer/pwendell.asc>
>>> 
>>> The documentation corresponding to this release can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/ <http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/>
>>> 
>>> The list of resolved issues are:
>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0 <https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0>
>>> 
>>> 
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Apache Spark workload and running on this candidate, then reporting
>>> any regressions.
>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> For additional commands, e-mail: dev-help@spark.apache.org <ma...@spark.apache.org>
>> 
> 
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Reynold Xin <rx...@databricks.com>.
You can find that by changing the filter to target version = 2.0.0. Cheers.

On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.marcu@inria.fr> wrote:

> +1 Great, I see the list of resolved issues, do you have a list of known
> issue you plan to stay with this release?
>
> with
> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
> -Phive-thriftserver -DskipTests clean package
>
> mvn -version
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T17:41:47+01:00)
> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
> Java version: 1.7.0_80, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [
> 2.635 s]
> [INFO] Spark Project Tags ................................. SUCCESS [
> 1.896 s]
> [INFO] Spark Project Sketch ............................... SUCCESS [
> 2.560 s]
> [INFO] Spark Project Networking ........................... SUCCESS [
> 6.533 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
> 4.176 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [
> 4.809 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [
> 6.242 s]
> [INFO] Spark Project Core ................................. SUCCESS [01:20
> min]
> [INFO] Spark Project GraphX ............................... SUCCESS [
> 9.148 s]
> [INFO] Spark Project Streaming ............................ SUCCESS [
> 22.760 s]
> [INFO] Spark Project Catalyst ............................. SUCCESS [
> 50.783 s]
> [INFO] Spark Project SQL .................................. SUCCESS [01:05
> min]
> [INFO] Spark Project ML Local Library ..................... SUCCESS [
> 4.281 s]
> [INFO] Spark Project ML Library ........................... SUCCESS [
> 54.537 s]
> [INFO] Spark Project Tools ................................ SUCCESS [
> 0.747 s]
> [INFO] Spark Project Hive ................................. SUCCESS [
> 33.032 s]
> [INFO] Spark Project HiveContext Compatibility ............ SUCCESS [
> 3.198 s]
> [INFO] Spark Project REPL ................................. SUCCESS [
> 3.573 s]
> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [
> 4.617 s]
> [INFO] Spark Project YARN ................................. SUCCESS [
> 7.321 s]
> [INFO] Spark Project Hive Thrift Server ................... SUCCESS [
> 16.496 s]
> [INFO] Spark Project Assembly ............................. SUCCESS [
> 2.300 s]
> [INFO] Spark Project External Flume Sink .................. SUCCESS [
> 4.219 s]
> [INFO] Spark Project External Flume ....................... SUCCESS [
> 6.987 s]
> [INFO] Spark Project External Flume Assembly .............. SUCCESS [
> 1.465 s]
> [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [
> 6.891 s]
> [INFO] Spark Project Examples ............................. SUCCESS [
> 13.465 s]
> [INFO] Spark Project External Kafka Assembly .............. SUCCESS [
> 2.815 s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 07:04 min
> [INFO] Finished at: 2016-05-18T17:55:33+02:00
> [INFO] Final Memory: 90M/824M
> [INFO]
> ------------------------------------------------------------------------
>
> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
>
> I think it's a good idea. Although releases have been preceded before
> by release candidates for developers, it would be good to get a formal
> preview/beta release ratified for public consumption ahead of a new
> major release. Better to have a little more testing in the wild to
> identify problems before 2.0.0 is finalized.
>
> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
> 8, compilation and tests succeed for "-Pyarn -Phive
> -Phive-thriftserver -Phadoop-2.6".
>
> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>
> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy
> (http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
> and passes if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is 2.0.0-preview
> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>
> The release files, including signatures, digests, etc. can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The documentation corresponding to this release can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>
> The list of resolved issues are:
>
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>
>
> If you are a Spark user, you can help us test this release by taking an
> existing Apache Spark workload and running on this candidate, then
> reporting
> any regressions.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Ovidiu-Cristian MARCU <ov...@inria.fr>.
+1 Great, I see the list of resolved issues, do you have a list of known issue you plan to stay with this release?

with
build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive -Phive-thriftserver -DskipTests clean package

mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
Maven home: /Users/omarcu/tools/apache-maven-3.3.9
Java version: 1.7.0_80, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"

[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.635 s]
[INFO] Spark Project Tags ................................. SUCCESS [  1.896 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  2.560 s]
[INFO] Spark Project Networking ........................... SUCCESS [  6.533 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  4.176 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  4.809 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  6.242 s]
[INFO] Spark Project Core ................................. SUCCESS [01:20 min]
[INFO] Spark Project GraphX ............................... SUCCESS [  9.148 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 22.760 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 50.783 s]
[INFO] Spark Project SQL .................................. SUCCESS [01:05 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [  4.281 s]
[INFO] Spark Project ML Library ........................... SUCCESS [ 54.537 s]
[INFO] Spark Project Tools ................................ SUCCESS [  0.747 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 33.032 s]
[INFO] Spark Project HiveContext Compatibility ............ SUCCESS [  3.198 s]
[INFO] Spark Project REPL ................................. SUCCESS [  3.573 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [  4.617 s]
[INFO] Spark Project YARN ................................. SUCCESS [  7.321 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 16.496 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  2.300 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [  4.219 s]
[INFO] Spark Project External Flume ....................... SUCCESS [  6.987 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [  1.465 s]
[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [  6.891 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 13.465 s]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  2.815 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:04 min
[INFO] Finished at: 2016-05-18T17:55:33+02:00
[INFO] Final Memory: 90M/824M
[INFO] ------------------------------------------------------------------------

> On 18 May 2016, at 16:28, Sean Owen <so...@cloudera.com> wrote:
> 
> I think it's a good idea. Although releases have been preceded before
> by release candidates for developers, it would be good to get a formal
> preview/beta release ratified for public consumption ahead of a new
> major release. Better to have a little more testing in the wild to
> identify problems before 2.0.0 is finalized.
> 
> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
> 8, compilation and tests succeed for "-Pyarn -Phive
> -Phive-thriftserver -Phadoop-2.6".
> 
> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
>> Hi,
>> 
>> In the past the Apache Spark community have created preview packages (not
>> official releases) and used those as opportunities to ask community members
>> to test the upcoming versions of Apache Spark. Several people in the Apache
>> community have suggested we conduct votes for these preview packages and
>> turn them into formal releases by the Apache foundation's standard. Preview
>> releases are not meant to be functional, i.e. they can and highly likely
>> will contain critical bugs or documentation errors, but we will be able to
>> post them to the project's website to get wider feedback. They should
>> satisfy the legal requirements of Apache's release policy
>> (http://www.apache.org/dev/release.html) such as having proper licenses.
>> 
>> 
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
>> and passes if a majority of at least 3 +1 PMC votes are cast.
>> 
>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>> [ ] -1 Do not release this package because ...
>> 
>> To learn more about Apache Spark, please see http://spark.apache.org/
>> 
>> The tag to be voted on is 2.0.0-preview
>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>> 
>> The release files, including signatures, digests, etc. can be found at:
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>> 
>> The documentation corresponding to this release can be found at:
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>> 
>> The list of resolved issues are:
>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>> 
>> 
>> If you are a Spark user, you can help us test this release by taking an
>> existing Apache Spark workload and running on this candidate, then reporting
>> any regressions.
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Sean Owen <so...@cloudera.com>.
I think it's a good idea. Although releases have been preceded before
by release candidates for developers, it would be good to get a formal
preview/beta release ratified for public consumption ahead of a new
major release. Better to have a little more testing in the wild to
identify problems before 2.0.0 is finalized.

+1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
8, compilation and tests succeed for "-Pyarn -Phive
-Phive-thriftserver -Phadoop-2.6".

On Wed, May 18, 2016 at 6:40 AM, Reynold Xin <rx...@apache.org> wrote:
> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy
> (http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
> and passes if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is 2.0.0-preview
> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>
> The release files, including signatures, digests, etc. can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The documentation corresponding to this release can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>
> The list of resolved issues are:
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>
>
> If you are a Spark user, you can help us test this release by taking an
> existing Apache Spark workload and running on this candidate, then reporting
> any regressions.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Hyukjin Kwon <gu...@gmail.com>.
I happened to test SparkR in Windows 7 (32 bits) and it seems some tests
are failed.

Could this be a reason to downvote?

For more details of the tests, please see
https://github.com/apache/spark/pull/13165#issuecomment-220515182



2016-05-20 13:44 GMT+09:00 Takuya UESHIN <ue...@happy-camper.st>:

> Hi all,
>
> I'm sorry, I misunderstood the purpose of this vote.
>
> I change to +1.
>
> Thanks.
>
>
>
>
> 2016-05-20 12:05 GMT+09:00 Takuya UESHIN <ue...@happy-camper.st>:
>
>> -1 (non-binding)
>>
>> I filed 2 major bugs of Spark SQL:
>>
>> SPARK-15308 <https://issues.apache.org/jira/browse/SPARK-15308>: RowEncoder
>> should preserve nested column name.
>> SPARK-15313 <https://issues.apache.org/jira/browse/SPARK-15313>: EmbedSerializerInFilter
>> rule should keep exprIds of output of surrounded SerializeFromObject.
>>
>> I've sent PRs for those, please check them.
>>
>> Thanks.
>>
>>
>>
>>
>> 2016-05-18 14:40 GMT+09:00 Reynold Xin <rx...@apache.org>:
>>
>>> Hi,
>>>
>>> In the past the Apache Spark community have created preview packages
>>> (not official releases) and used those as opportunities to ask community
>>> members to test the upcoming versions of Apache Spark. Several people in
>>> the Apache community have suggested we conduct votes for these preview
>>> packages and turn them into formal releases by the Apache foundation's
>>> standard. Preview releases are not meant to be functional, i.e. they can
>>> and highly likely will contain critical bugs or documentation errors, but
>>> we will be able to post them to the project's website to get wider
>>> feedback. They should satisfy the legal requirements of Apache's release
>>> policy (http://www.apache.org/dev/release.html) such as having proper
>>> licenses.
>>>
>>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
>>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is 2.0.0-preview
>>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>>
>>> The list of resolved issues are:
>>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>>
>>>
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Apache Spark workload and running on this candidate, then
>>> reporting any regressions.
>>>
>>>
>>
>>
>> --
>> Takuya UESHIN
>> Tokyo, Japan
>>
>> http://twitter.com/ueshin
>>
>
>
>
> --
> Takuya UESHIN
> Tokyo, Japan
>
> http://twitter.com/ueshin
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Takuya UESHIN <ue...@happy-camper.st>.
Hi all,

I'm sorry, I misunderstood the purpose of this vote.

I change to +1.

Thanks.




2016-05-20 12:05 GMT+09:00 Takuya UESHIN <ue...@happy-camper.st>:

> -1 (non-binding)
>
> I filed 2 major bugs of Spark SQL:
>
> SPARK-15308 <https://issues.apache.org/jira/browse/SPARK-15308>: RowEncoder
> should preserve nested column name.
> SPARK-15313 <https://issues.apache.org/jira/browse/SPARK-15313>: EmbedSerializerInFilter
> rule should keep exprIds of output of surrounded SerializeFromObject.
>
> I've sent PRs for those, please check them.
>
> Thanks.
>
>
>
>
> 2016-05-18 14:40 GMT+09:00 Reynold Xin <rx...@apache.org>:
>
>> Hi,
>>
>> In the past the Apache Spark community have created preview packages (not
>> official releases) and used those as opportunities to ask community members
>> to test the upcoming versions of Apache Spark. Several people in the Apache
>> community have suggested we conduct votes for these preview packages and
>> turn them into formal releases by the Apache foundation's standard. Preview
>> releases are not meant to be functional, i.e. they can and highly likely
>> will contain critical bugs or documentation errors, but we will be able to
>> post them to the project's website to get wider feedback. They should
>> satisfy the legal requirements of Apache's release policy (
>> http://www.apache.org/dev/release.html) such as having proper licenses.
>>
>>
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
>> and passes if a majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is 2.0.0-preview
>> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The documentation corresponding to this release can be found at:
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>>
>> The list of resolved issues are:
>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>>
>>
>> If you are a Spark user, you can help us test this release by taking an
>> existing Apache Spark workload and running on this candidate, then
>> reporting any regressions.
>>
>>
>
>
> --
> Takuya UESHIN
> Tokyo, Japan
>
> http://twitter.com/ueshin
>



-- 
Takuya UESHIN
Tokyo, Japan

http://twitter.com/ueshin

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Posted by Takuya UESHIN <ue...@happy-camper.st>.
-1 (non-binding)

I filed 2 major bugs of Spark SQL:

SPARK-15308 <https://issues.apache.org/jira/browse/SPARK-15308>: RowEncoder
should preserve nested column name.
SPARK-15313 <https://issues.apache.org/jira/browse/SPARK-15313>:
EmbedSerializerInFilter
rule should keep exprIds of output of surrounded SerializeFromObject.

I've sent PRs for those, please check them.

Thanks.




2016-05-18 14:40 GMT+09:00 Reynold Xin <rx...@apache.org>:

> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy (
> http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
> and passes if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is 2.0.0-preview
> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>
> The release files, including signatures, digests, etc. can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The documentation corresponding to this release can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>
> The list of resolved issues are:
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>
>
> If you are a Spark user, you can help us test this release by taking an
> existing Apache Spark workload and running on this candidate, then
> reporting any regressions.
>
>


-- 
Takuya UESHIN
Tokyo, Japan

http://twitter.com/ueshin