You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ShuaiW <gi...@git.apache.org> on 2017/07/23 18:41:41 UTC

[GitHub] spark pull request #18720: Spark 2.2.0: pip pyspark doesn't work well in not...

GitHub user ShuaiW opened a pull request:

    https://github.com/apache/spark/pull/18720

    Spark 2.2.0: pip pyspark doesn't work well in notebook on Windows 10; got Exception: Java gateway process exited before sending the driver its port number

    - Step1: `pip install pyspark`
    - Step 2: `jupyter notebook`
    - Step 3: initialize a Spark Context
    
    ```python
    from pyspark import SparkContext
    sc = SparkContext()
    ```
    But I got the error `Exception: Java gateway process exited before sending the driver its port number`. 
    
    One workaround is to point SPARK_HOME variable to one of the pre-built Spark package directory (e.g., `C:\spark-2.2.0-bin-hadoop2.7`), and step 3 works fine so a Spark Context is initialized. I looked through the pyspark folder installed by pip and the pre-built package from download and found the file structures are a bit different, which should be the cause for the problem.
    
    Weirdly, step 1-3 seems to work well on Mac OS 10.11.6.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/spark branch-2.2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18720.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18720
    
----
commit 6c5b7e106895302a87cf6522d3c64c3badac699f
Author: Felix Cheung <fe...@hotmail.com>
Date:   2017-05-08T06:10:18Z

    [SPARK-20626][SPARKR] address date test warning with timezone on windows
    
    ## What changes were proposed in this pull request?
    
    set timezone on windows
    
    ## How was this patch tested?
    
    unit test, AppVeyor
    
    Author: Felix Cheung <fe...@hotmail.com>
    
    Closes #17892 from felixcheung/rtimestamptest.
    
    (cherry picked from commit c24bdaab5a234d18b273544cefc44cc4005bf8fc)
    Signed-off-by: Felix Cheung <fe...@apache.org>

commit d8a5a0d3420abbb911d8a80dc7165762eb08d779
Author: Wayne Zhang <ac...@uber.com>
Date:   2017-05-08T06:16:30Z

    [SPARKR][DOC] fix typo in vignettes
    
    ## What changes were proposed in this pull request?
    Fix typo in vignettes
    
    Author: Wayne Zhang <ac...@uber.com>
    
    Closes #17884 from actuaryzhang/typo.
    
    (cherry picked from commit 2fdaeb52bbe2ed1a9127ac72917286e505303c85)
    Signed-off-by: Felix Cheung <fe...@apache.org>

commit 7b9d05ad00455daa53ae4ef1a602a6c64c2c95a4
Author: Nick Pentreath <ni...@za.ibm.com>
Date:   2017-05-08T10:45:00Z

    [SPARK-20596][ML][TEST] Consolidate and improve ALS recommendAll test cases
    
    Existing test cases for `recommendForAllX` methods (added in [SPARK-19535](https://issues.apache.org/jira/browse/SPARK-19535)) test `k < num items` and `k = num items`. Technically we should also test that `k > num items` returns the same results as `k = num items`.
    
    ## How was this patch tested?
    
    Updated existing unit tests.
    
    Author: Nick Pentreath <ni...@za.ibm.com>
    
    Closes #17860 from MLnick/SPARK-20596-als-rec-tests.
    
    (cherry picked from commit 58518d070777fc0665c4d02bad8adf910807df98)
    Signed-off-by: Nick Pentreath <ni...@za.ibm.com>

commit 23681e9ca0042328f93962701d19ca371727b0b7
Author: Xianyang Liu <xi...@intel.com>
Date:   2017-05-08T17:25:24Z

    [SPARK-20621][DEPLOY] Delete deprecated config parameter in 'spark-env.sh'
    
    ## What changes were proposed in this pull request?
    
    Currently, `spark.executor.instances` is deprecated in `spark-env.sh`, because we suggest config it in `spark-defaults.conf` or other config file. And also this parameter is useless even if you set it in `spark-env.sh`, so remove it in this patch.
    
    ## How was this patch tested?
    
    Existing tests.
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.
    
    Author: Xianyang Liu <xi...@intel.com>
    
    Closes #17881 from ConeyLiu/deprecatedParam.
    
    (cherry picked from commit aeb2ecc0cd898f5352df0a04be1014b02ea3e20e)
    Signed-off-by: Marcelo Vanzin <va...@cloudera.com>

commit 4179ffc031a0dbca6a93255c673de800ce7393fe
Author: Hossein <ho...@databricks.com>
Date:   2017-05-08T21:48:11Z

    [SPARK-20661][SPARKR][TEST] SparkR tableNames() test fails
    
    ## What changes were proposed in this pull request?
    Cleaning existing temp tables before running tableNames tests
    
    ## How was this patch tested?
    SparkR Unit tests
    
    Author: Hossein <ho...@databricks.com>
    
    Closes #17903 from falaki/SPARK-20661.
    
    (cherry picked from commit 2abfee18b6511482b916c36f00bf3abf68a59e19)
    Signed-off-by: Yin Huai <yh...@databricks.com>

commit 54e07434968624dbb0fb80773356e614b954e52f
Author: Felix Cheung <fe...@hotmail.com>
Date:   2017-05-09T05:49:40Z

    [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames() test fails
    
    ## What changes were proposed in this pull request?
    
    Change it to check for relative count like in this test https://github.com/apache/spark/blame/master/R/pkg/inst/tests/testthat/test_sparkSQL.R#L3355 for catalog APIs
    
    ## How was this patch tested?
    
    unit tests, this needs to combine with another commit with SQL change to check
    
    Author: Felix Cheung <fe...@hotmail.com>
    
    Closes #17905 from felixcheung/rtabletests.
    
    (cherry picked from commit b952b44af4d243f1e3ad88bccf4af7d04df3fc81)
    Signed-off-by: Felix Cheung <fe...@apache.org>

commit 72fca9a0a7a6dd2ab7c338fab9666b51cd981cce
Author: Peng <pe...@intel.com>
Date:   2017-05-09T08:05:49Z

    [SPARK-11968][MLLIB] Optimize MLLIB ALS recommendForAll
    
    The recommendForAll of MLLIB ALS is very slow.
    GC is a key problem of the current method.
    The task use the following code to keep temp result:
    val output = new Array[(Int, (Int, Double))](m*n)
    m = n = 4096 (default value, no method to set)
    so output is about 4k * 4k * (4 + 4 + 8) = 256M. This is a large memory and cause serious GC problem, and it is frequently OOM.
    
    Actually, we don't need to save all the temp result. Support we recommend topK (topK is about 10, or 20) product for each user, we only need 4k * topK * (4 + 4 + 8) memory to save the temp result.
    
    The Test Environment:
    3 workers: each work 10 core, each work 30G memory, each work 1 executor.
    The Data: User 480,000, and Item 17,000
    
    BlockSize:     1024  2048  4096  8192
    Old method:  245s  332s  488s  OOM
    This solution: 121s  118s   117s  120s
    
    The existing UT.
    
    Author: Peng <pe...@intel.com>
    Author: Peng Meng <pe...@intel.com>
    
    Closes #17742 from mpjlu/OptimizeAls.
    
    (cherry picked from commit 8079424763c2043264f30a6898ce964379bd9b56)
    Signed-off-by: Nick Pentreath <ni...@za.ibm.com>

commit ca3f7edbad6a2e7fcd1c1d3dbd1a522cd0d7c476
Author: Nick Pentreath <ni...@za.ibm.com>
Date:   2017-05-09T08:13:15Z

    [SPARK-20587][ML] Improve performance of ML ALS recommendForAll
    
    This PR is a `DataFrame` version of #17742 for [SPARK-11968](https://issues.apache.org/jira/browse/SPARK-11968), for improving the performance of `recommendAll` methods.
    
    ## How was this patch tested?
    
    Existing unit tests.
    
    Author: Nick Pentreath <ni...@za.ibm.com>
    
    Closes #17845 from MLnick/ml-als-perf.
    
    (cherry picked from commit 10b00abadf4a3473332eef996db7b66f491316f2)
    Signed-off-by: Nick Pentreath <ni...@za.ibm.com>

commit 4bbfad44e426365ad9f4941d68c110523b17ea6d
Author: Jon McLean <jo...@atsid.com>
Date:   2017-05-09T08:47:50Z

    [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException
    
    ## What changes were proposed in this pull request?
    
    Added a check for for the number of defined values.  Previously the argmax function assumed that at least one value was defined if the vector size was greater than zero.
    
    ## How was this patch tested?
    
    Tests were added to the existing VectorsSuite to cover this case.
    
    Author: Jon McLean <jo...@atsid.com>
    
    Closes #17877 from jonmclean/vectorArgmaxIndexBug.
    
    (cherry picked from commit be53a78352ae7c70d8a07d0df24574b3e3129b4a)
    Signed-off-by: Sean Owen <so...@cloudera.com>

commit 4b7aa0b1dbd85e2238acba45e8f94c097358fb72
Author: Yanbo Liang <yb...@gmail.com>
Date:   2017-05-09T09:30:37Z

    [SPARK-20606][ML] ML 2.2 QA: Remove deprecated methods for ML
    
    ## What changes were proposed in this pull request?
    Remove ML methods we deprecated in 2.1.
    
    ## How was this patch tested?
    Existing tests.
    
    Author: Yanbo Liang <yb...@gmail.com>
    
    Closes #17867 from yanboliang/spark-20606.
    
    (cherry picked from commit b8733e0ad9f5a700f385e210450fd2c10137293e)
    Signed-off-by: Yanbo Liang <yb...@gmail.com>

commit b3309676bb83a80d38b916066d046866a6f42ef0
Author: Xiao Li <ga...@gmail.com>
Date:   2017-05-09T12:10:50Z

    [SPARK-20667][SQL][TESTS] Cleanup the cataloged metadata after completing the package of sql/core and sql/hive
    
    ## What changes were proposed in this pull request?
    
    So far, we do not drop all the cataloged objects after each package. Sometimes, we might hit strange test case errors because the previous test suite did not drop the cataloged/temporary objects (tables/functions/database). At least, we can first clean up the environment when completing the package of `sql/core` and `sql/hive`.
    
    ## How was this patch tested?
    N/A
    
    Author: Xiao Li <ga...@gmail.com>
    
    Closes #17908 from gatorsmile/reset.
    
    (cherry picked from commit 0d00c768a860fc03402c8f0c9081b8147c29133e)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit 272d2a10d70588e1f80cc6579d4ec3c44b5bbfc2
Author: Takeshi Yamamuro <ya...@apache.org>
Date:   2017-05-09T12:22:51Z

    [SPARK-20311][SQL] Support aliases for table value functions
    
    ## What changes were proposed in this pull request?
    This pr added parsing rules to support aliases in table value functions.
    
    ## How was this patch tested?
    Added tests in `PlanParserSuite`.
    
    Author: Takeshi Yamamuro <ya...@apache.org>
    
    Closes #17666 from maropu/SPARK-20311.
    
    (cherry picked from commit 714811d0b5bcb5d47c39782ff74f898d276ecc59)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit 08e1b78f01955c7151d9e984d392d45deced6e34
Author: Wenchen Fan <we...@databricks.com>
Date:   2017-05-09T16:09:35Z

    [SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases
    
    `ReplSuite.newProductSeqEncoder with REPL defined class` was flaky and throws OOM exception frequently. By analyzing the heap dump, we found the reason is that, in each test case of `ReplSuite`, we create a REPL instance, which creates a classloader and loads a lot of classes related to `SparkContext`. More details please see https://github.com/apache/spark/pull/17833#issuecomment-298711435.
    
    In this PR, we create a new test suite, `SingletonReplSuite`, which shares one REPL instances among all the test cases. Then we move most of the tests from `ReplSuite` to `SingletonReplSuite`, to avoid creating a lot of REPL instances and reduce memory footprint.
    
    test only change
    
    Author: Wenchen Fan <we...@databricks.com>
    
    Closes #17844 from cloud-fan/flaky-test.
    
    (cherry picked from commit f561a76b2f895dea52f228a9376948242c3331ad)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit 73aa23b8ef64960e7f171aa07aec396667a2339d
Author: Reynold Xin <rx...@databricks.com>
Date:   2017-05-09T16:24:28Z

    [SPARK-20674][SQL] Support registering UserDefinedFunction as named UDF
    
    ## What changes were proposed in this pull request?
    For some reason we don't have an API to register UserDefinedFunction as named UDF. It is a no brainer to add one, in addition to the existing register functions we have.
    
    ## How was this patch tested?
    Added a test case in UDFSuite for the new API.
    
    Author: Reynold Xin <rx...@databricks.com>
    
    Closes #17915 from rxin/SPARK-20674.
    
    (cherry picked from commit d099f414d2cb53f5a61f6e77317c736be6f953a0)
    Signed-off-by: Xiao Li <ga...@gmail.com>

commit c7bd909f67209b4d1354c3d5b0a0fb1d4e28f205
Author: Sean Owen <so...@cloudera.com>
Date:   2017-05-09T17:22:23Z

    [SPARK-19876][BUILD] Move Trigger.java to java source hierarchy
    
    ## What changes were proposed in this pull request?
    
    Simply moves `Trigger.java` to `src/main/java` from `src/main/scala`
    See https://github.com/apache/spark/pull/17219
    
    ## How was this patch tested?
    
    Existing tests.
    
    Author: Sean Owen <so...@cloudera.com>
    
    Closes #17921 from srowen/SPARK-19876.2.
    
    (cherry picked from commit 25ee816e090c42f0e35be2d2cb0f8ec60726317c)
    Signed-off-by: Herman van Hovell <hv...@databricks.com>

commit 9e8d23b3a2f99985ffb3c4eb67ac0a2774fa5b02
Author: Holden Karau <ho...@us.ibm.com>
Date:   2017-05-09T18:25:29Z

    [SPARK-20627][PYSPARK] Drop the hadoop distirbution name from the Python version
    
    ## What changes were proposed in this pull request?
    
    Drop the hadoop distirbution name from the Python version (PEP440 - https://www.python.org/dev/peps/pep-0440/). We've been using the local version string to disambiguate between different hadoop versions packaged with PySpark, but PEP0440 states that local versions should not be used when publishing up-stream. Since we no longer make PySpark pip packages for different hadoop versions, we can simply drop the hadoop information. If at a later point we need to start publishing different hadoop versions we can look at make different packages or similar.
    
    ## How was this patch tested?
    
    Ran `make-distribution` locally
    
    Author: Holden Karau <ho...@us.ibm.com>
    
    Closes #17885 from holdenk/SPARK-20627-remove-pip-local-version-string.
    
    (cherry picked from commit 1b85bcd9298cf84dd746fe8e91ab0b0df69ef17e)
    Signed-off-by: Holden Karau <ho...@us.ibm.com>

commit d191b962dc81c015fa92a38d882a8c7ea620ef06
Author: Yin Huai <yh...@databricks.com>
Date:   2017-05-09T21:47:45Z

    Revert "[SPARK-20311][SQL] Support aliases for table value functions"
    
    This reverts commit 714811d0b5bcb5d47c39782ff74f898d276ecc59.

commit 7600a7ab65777a59f3a33edef40328b6a5d864ef
Author: uncleGen <hu...@gmail.com>
Date:   2017-05-09T22:08:09Z

    [SPARK-20373][SQL][SS] Batch queries with 'Dataset/DataFrame.withWatermark()` does not execute
    
    ## What changes were proposed in this pull request?
    
    Any Dataset/DataFrame batch query with the operation `withWatermark` does not execute because the batch planner does not have any rule to explicitly handle the EventTimeWatermark logical plan.
    The right solution is to simply remove the plan node, as the watermark should not affect any batch query in any way.
    
    Changes:
    - In this PR, we add a new rule `EliminateEventTimeWatermark` to check if we need to ignore the event time watermark. We will ignore watermark in any batch query.
    
    Depends upon:
    - [SPARK-20672](https://issues.apache.org/jira/browse/SPARK-20672). We can not add this rule into analyzer directly, because streaming query will be copied to `triggerLogicalPlan ` in every trigger, and the rule will be applied to `triggerLogicalPlan` mistakenly.
    
    Others:
    - A typo fix in example.
    
    ## How was this patch tested?
    
    add new unit test.
    
    Author: uncleGen <hu...@gmail.com>
    
    Closes #17896 from uncleGen/SPARK-20373.
    
    (cherry picked from commit c0189abc7c6ddbecc1832d2ff0cfc5546a010b60)
    Signed-off-by: Shixiong Zhu <sh...@databricks.com>

commit 6a996b36283dcd22ff7aa38968a80f575d2f151e
Author: Yuming Wang <wg...@gmail.com>
Date:   2017-05-10T02:45:00Z

    [SPARK-17685][SQL] Make SortMergeJoinExec's currentVars is null when calling createJoinKey
    
    ## What changes were proposed in this pull request?
    
    The following SQL query cause `IndexOutOfBoundsException` issue when `LIMIT > 1310720`:
    ```sql
    CREATE TABLE tab1(int int, int2 int, str string);
    CREATE TABLE tab2(int int, int2 int, str string);
    INSERT INTO tab1 values(1,1,'str');
    INSERT INTO tab1 values(2,2,'str');
    INSERT INTO tab2 values(1,1,'str');
    INSERT INTO tab2 values(2,3,'str');
    
    SELECT
      count(*)
    FROM
      (
        SELECT t1.int, t2.int2
        FROM (SELECT * FROM tab1 LIMIT 1310721) t1
        INNER JOIN (SELECT * FROM tab2 LIMIT 1310721) t2
        ON (t1.int = t2.int AND t1.int2 = t2.int2)
      ) t;
    ```
    
    This pull request fix this issue.
    
    ## How was this patch tested?
    
    unit tests
    
    Author: Yuming Wang <wg...@gmail.com>
    
    Closes #17920 from wangyum/SPARK-17685.
    
    (cherry picked from commit 771abeb46f637592aba2e63db2ed05b6cabfd0be)
    Signed-off-by: Herman van Hovell <hv...@databricks.com>

commit 7b6f3a118e973216264bbf356af2bb1e7870466e
Author: hyukjinkwon <gu...@gmail.com>
Date:   2017-05-10T05:44:47Z

    [SPARK-20590][SQL] Use Spark internal datasource if multiples are found for the same shorten name
    
    ## What changes were proposed in this pull request?
    
    One of the common usability problems around reading data in spark (particularly CSV) is that there can often be a conflict between different readers in the classpath.
    
    As an example, if someone launches a 2.x spark shell with the spark-csv package in the classpath, Spark currently fails in an extremely unfriendly way (see databricks/spark-csv#367):
    
    ```bash
    ./bin/spark-shell --packages com.databricks:spark-csv_2.11:1.5.0
    scala> val df = spark.read.csv("/foo/bar.csv")
    java.lang.RuntimeException: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat, com.databricks.spark.csv.DefaultSource15), please specify the fully qualified class name.
      at scala.sys.package$.error(package.scala:27)
      at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:574)
      at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:85)
      at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:85)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:295)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:533)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:412)
      ... 48 elided
    ```
    
    This PR proposes a simple way of fixing this error by picking up the internal datasource if there is single (the datasource that has "org.apache.spark" prefix).
    
    ```scala
    scala> spark.range(1).write.format("csv").mode("overwrite").save("/tmp/abc")
    17/05/10 09:47:44 WARN DataSource: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat,
    com.databricks.spark.csv.DefaultSource15), defaulting to the internal datasource (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat).
    ```
    
    ```scala
    scala> spark.range(1).write.format("Csv").mode("overwrite").save("/tmp/abc")
    17/05/10 09:47:52 WARN DataSource: Multiple sources found for Csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat,
    com.databricks.spark.csv.DefaultSource15), defaulting to the internal datasource (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat).
    ```
    
    ## How was this patch tested?
    
    Manually tested as below:
    
    ```bash
    ./bin/spark-shell --packages com.databricks:spark-csv_2.11:1.5.0
    ```
    
    ```scala
    spark.sparkContext.setLogLevel("WARN")
    ```
    
    **positive cases**:
    
    ```scala
    scala> spark.range(1).write.format("csv").mode("overwrite").save("/tmp/abc")
    17/05/10 09:47:44 WARN DataSource: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat,
    com.databricks.spark.csv.DefaultSource15), defaulting to the internal datasource (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat).
    ```
    
    ```scala
    scala> spark.range(1).write.format("Csv").mode("overwrite").save("/tmp/abc")
    17/05/10 09:47:52 WARN DataSource: Multiple sources found for Csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat,
    com.databricks.spark.csv.DefaultSource15), defaulting to the internal datasource (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat).
    ```
    
    (newlines were inserted for readability).
    
    ```scala
    scala> spark.range(1).write.format("com.databricks.spark.csv").mode("overwrite").save("/tmp/abc")
    ```
    
    ```scala
    scala> spark.range(1).write.format("org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").mode("overwrite").save("/tmp/abc")
    ```
    
    **negative cases**:
    
    ```scala
    scala> spark.range(1).write.format("com.databricks.spark.csv.CsvRelation").save("/tmp/abc")
    java.lang.InstantiationException: com.databricks.spark.csv.CsvRelation
    ...
    ```
    
    ```scala
    scala> spark.range(1).write.format("com.databricks.spark.csv.CsvRelatio").save("/tmp/abc")
    java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv.CsvRelatio. Please find packages at http://spark.apache.org/third-party-projects.html
    ...
    ```
    
    Author: hyukjinkwon <gu...@gmail.com>
    
    Closes #17916 from HyukjinKwon/datasource-detect.
    
    (cherry picked from commit 3d2131ab4ddead29601fb3c597b798202ac25fdd)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit ef50a954882fa1911f7ede3f0aefc8fcf09c6059
Author: Josh Rosen <jo...@databricks.com>
Date:   2017-05-10T06:36:36Z

    [SPARK-20686][SQL] PropagateEmptyRelation incorrectly handles aggregate without grouping
    
    ## What changes were proposed in this pull request?
    
    The query
    
    ```
    SELECT 1 FROM (SELECT COUNT(*) WHERE FALSE) t1
    ```
    
    should return a single row of output because the subquery is an aggregate without a group-by and thus should return a single row. However, Spark incorrectly returns zero rows.
    
    This is caused by SPARK-16208 / #13906, a patch which added an optimizer rule to propagate EmptyRelation through operators. The logic for handling aggregates is wrong: it checks whether aggregate expressions are non-empty for deciding whether the output should be empty, whereas it should be checking grouping expressions instead:
    
    An aggregate with non-empty grouping expression will return one output row per group. If the input to the grouped aggregate is empty then all groups will be empty and thus the output will be empty. It doesn't matter whether the aggregation output columns include aggregate expressions since that won't affect the number of output rows.
    
    If the grouping expressions are empty, however, then the aggregate will always produce a single output row and thus we cannot propagate the EmptyRelation.
    
    The current implementation is incorrect and also misses an optimization opportunity by not propagating EmptyRelation in the case where a grouped aggregate has aggregate expressions (in other words, `SELECT COUNT(*) from emptyRelation GROUP BY x` would _not_ be optimized to `EmptyRelation` in the old code, even though it safely could be).
    
    This patch resolves this issue by modifying `PropagateEmptyRelation` to consider only the presence/absence of grouping expressions, not the aggregate functions themselves, when deciding whether to propagate EmptyRelation.
    
    ## How was this patch tested?
    
    - Added end-to-end regression tests in `SQLQueryTest`'s `group-by.sql` file.
    - Updated unit tests in `PropagateEmptyRelationSuite`.
    
    Author: Josh Rosen <jo...@databricks.com>
    
    Closes #17929 from JoshRosen/fix-PropagateEmptyRelation.
    
    (cherry picked from commit a90c5cd8226146a58362732171b92cb99a7bc4c7)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit 3ed2f4d516ce02dfef929195778f8214703913d8
Author: zero323 <ze...@users.noreply.github.com>
Date:   2017-05-10T08:57:52Z

    [SPARK-20631][PYTHON][ML] LogisticRegression._checkThresholdConsistency should use values not Params
    
    ## What changes were proposed in this pull request?
    
    - Replace `getParam` calls with `getOrDefault` calls.
    - Fix exception message to avoid unintended `TypeError`.
    - Add unit tests
    
    ## How was this patch tested?
    
    New unit tests.
    
    Author: zero323 <ze...@users.noreply.github.com>
    
    Closes #17891 from zero323/SPARK-20631.
    
    (cherry picked from commit 804949c6bf00b8e26c39d48bbcc4d0470ee84e47)
    Signed-off-by: Yanbo Liang <yb...@gmail.com>

commit 7597a522b7e5be43910e86cd6f805e7e9ee08ced
Author: Alex Bozarth <aj...@us.ibm.com>
Date:   2017-05-10T09:20:10Z

    [SPARK-20630][WEB UI] Fixed column visibility in Executor Tab
    
    ## What changes were proposed in this pull request?
    
    #14617 added new columns to the executor table causing the visibility checks for the logs and threadDump columns to toggle the wrong columns since they used hard-coded column numbers.
    
    I've updated the checks to use column names instead of numbers so future updates don't accidentally break this again.
    
    Note: This will also need to be back ported into 2.2 since #14617 was merged there
    
    ## How was this patch tested?
    
    Manually tested
    
    Author: Alex Bozarth <aj...@us.ibm.com>
    
    Closes #17904 from ajbozarth/spark20630.
    
    (cherry picked from commit ca4625e0e58df7f02346470d22a9478d9640709d)
    Signed-off-by: Sean Owen <so...@cloudera.com>

commit 0851b6cfb8980fa8816a96026fbf0498799e296b
Author: Wenchen Fan <we...@databricks.com>
Date:   2017-05-10T11:30:00Z

    [SPARK-20688][SQL] correctly check analysis for scalar sub-queries
    
    ## What changes were proposed in this pull request?
    
    In `CheckAnalysis`, we should call `checkAnalysis` for `ScalarSubquery` at the beginning, as later we will call `plan.output` which is invalid if `plan` is not resolved.
    
    ## How was this patch tested?
    
    new regression test
    
    Author: Wenchen Fan <we...@databricks.com>
    
    Closes #17930 from cloud-fan/tmp.
    
    (cherry picked from commit 789bdbe3d0d9558043872161bdfa148ec021a849)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit 5f6029c7500b0c5a769c6b62879d8532a5692a50
Author: wangzhenhua <wa...@huawei.com>
Date:   2017-05-10T11:42:49Z

    [SPARK-20678][SQL] Ndv for columns not in filter condition should also be updated
    
    ## What changes were proposed in this pull request?
    
    In filter estimation, we update column stats for those columns in filter condition. However, if the number of rows decreases after the filter (i.e. the overall selectivity is less than 1), we need to update (scale down) the number of distinct values (NDV) for all columns, no matter they are in filter conditions or not.
    
    This pr also fixes the inconsistency of rounding mode for ndv and rowCount.
    
    ## How was this patch tested?
    
    Added new tests.
    
    Author: wangzhenhua <wa...@huawei.com>
    
    Closes #17918 from wzhfy/scaleDownNdvAfterFilter.
    
    (cherry picked from commit 76e4a5566b1e9579632e03440cecd04dd142bc44)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

commit 358516dcbef5178cdc6cb4387d7f6837359946ba
Author: Xianyang Liu <xi...@intel.com>
Date:   2017-05-10T12:56:34Z

    [MINOR][BUILD] Fix lint-java breaks.
    
    ## What changes were proposed in this pull request?
    
    This PR proposes to fix the lint-breaks as below:
    ```
    [ERROR] src/main/java/org/apache/spark/unsafe/Platform.java:[51] (regexp) RegexpSingleline: No trailing whitespace allowed.
    [ERROR] src/main/scala/org/apache/spark/sql/streaming/Trigger.java:[45,25] (naming) MethodName: Method name 'ProcessingTime' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.
    [ERROR] src/main/scala/org/apache/spark/sql/streaming/Trigger.java:[62,25] (naming) MethodName: Method name 'ProcessingTime' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.
    [ERROR] src/main/scala/org/apache/spark/sql/streaming/Trigger.java:[78,25] (naming) MethodName: Method name 'ProcessingTime' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.
    [ERROR] src/main/scala/org/apache/spark/sql/streaming/Trigger.java:[92,25] (naming) MethodName: Method name 'ProcessingTime' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.
    [ERROR] src/main/scala/org/apache/spark/sql/streaming/Trigger.java:[102,25] (naming) MethodName: Method name 'Once' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.
    [ERROR] src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java:[28,8] (imports) UnusedImports: Unused import - org.apache.spark.streaming.api.java.JavaDStream.
    ```
    
    after:
    ```
    dev/lint-java
    Checkstyle checks passed.
    ```
    [Test Result](https://travis-ci.org/ConeyLiu/spark/jobs/229666169)
    
    ## How was this patch tested?
    
    Travis CI
    
    Author: Xianyang Liu <xi...@intel.com>
    
    Closes #17890 from ConeyLiu/codestyle.
    
    (cherry picked from commit fcb88f9211e39c705073db5300c96ceeb3f227d7)
    Signed-off-by: Sean Owen <so...@cloudera.com>

commit 86cef4df5fd9e28a8ece4ec33376d3622de2ef69
Author: Ala Luszczak <al...@databricks.com>
Date:   2017-05-10T15:41:04Z

    [SPARK-19447] Remove remaining references to generated rows metric
    
    ## What changes were proposed in this pull request?
    
    https://github.com/apache/spark/commit/b486ffc86d8ad6c303321dcf8514afee723f61f8 left behind references to "number of generated rows" metrics, that should have been removed.
    
    ## How was this patch tested?
    
    Existing unit tests.
    
    Author: Ala Luszczak <al...@databricks.com>
    
    Closes #17939 from ala/SPARK-19447-fix.
    
    (cherry picked from commit 5c2c4dcce529d228a97ede0386b95213ea0e1da5)
    Signed-off-by: Herman van Hovell <hv...@databricks.com>

commit 3eb0ee06a588da5b9c08a72d178835c6e8bad36b
Author: Josh Rosen <jo...@databricks.com>
Date:   2017-05-10T23:50:57Z

    [SPARK-20685] Fix BatchPythonEvaluation bug in case of single UDF w/ repeated arg.
    
    ## What changes were proposed in this pull request?
    
    There's a latent corner-case bug in PySpark UDF evaluation where executing a `BatchPythonEvaluation` with a single multi-argument UDF where _at least one argument value is repeated_ will crash at execution with a confusing error.
    
    This problem was introduced in #12057: the code there has a fast path for handling a "batch UDF evaluation consisting of a single Python UDF", but that branch incorrectly assumes that a single UDF won't have repeated arguments and therefore skips the code for unpacking arguments from the input row (whose schema may not necessarily match the UDF inputs due to de-duplication of repeated arguments which occurred in the JVM before sending UDF inputs to Python).
    
    This fix here is simply to remove this special-casing: it turns out that the code in the "multiple UDFs" branch just so happens to work for the single-UDF case because Python treats `(x)` as equivalent to `x`, not as a single-argument tuple.
    
    ## How was this patch tested?
    
    New regression test in `pyspark.python.sql.tests` module (tested and confirmed that it fails before my fix).
    
    Author: Josh Rosen <jo...@databricks.com>
    
    Closes #17927 from JoshRosen/SPARK-20685.
    
    (cherry picked from commit 8ddbc431d8b21d5ee57d3d209a4f25e301f15283)
    Signed-off-by: Xiao Li <ga...@gmail.com>

commit 80a57fa90be8dca4340345c09b4ea28fbf11a516
Author: Yanbo Liang <yb...@gmail.com>
Date:   2017-05-11T06:48:13Z

    [SPARK-20606][ML] Revert "[] ML 2.2 QA: Remove deprecated methods for ML"
    
    This reverts commit b8733e0ad9f5a700f385e210450fd2c10137293e.
    
    Author: Yanbo Liang <yb...@gmail.com>
    
    Closes #17944 from yanboliang/spark-20606-revert.
    
    (cherry picked from commit 0698e6c88ca11fdfd6e5498cab784cf6dbcdfacb)
    Signed-off-by: Yanbo Liang <yb...@gmail.com>

commit dd9e3b2c976a4ef3b4837590a2ba0954cf73860d
Author: Wenchen Fan <we...@databricks.com>
Date:   2017-05-11T07:41:15Z

    [SPARK-20569][SQL] RuntimeReplaceable functions should not take extra parameters
    
    ## What changes were proposed in this pull request?
    
    `RuntimeReplaceable` always has a constructor with the expression to replace with, and this constructor should not be the function builder.
    
    ## How was this patch tested?
    
    new regression test
    
    Author: Wenchen Fan <we...@databricks.com>
    
    Closes #17876 from cloud-fan/minor.
    
    (cherry picked from commit b4c99f43690f8cfba414af90fa2b3998a510bba8)
    Signed-off-by: Xiao Li <ga...@gmail.com>

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18720: Spark 2.2.0: pip pyspark doesn't work well in notebook o...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18720
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18720: Spark 2.2.0: pip pyspark doesn't work well in notebook o...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/18720
  
    Could you close it and resubmit the PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18720: Spark 2.2.0: pip pyspark doesn't work well in not...

Posted by ShuaiW <gi...@git.apache.org>.
Github user ShuaiW closed the pull request at:

    https://github.com/apache/spark/pull/18720


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18720: Spark 2.2.0: pip pyspark doesn't work well in notebook o...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18720
  
    @ShuaiW close this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org