You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2018/08/23 09:53:17 UTC

[GitHub] spark pull request #22198: [SPARK-25121][SQL] Supports multi-part table name...

GitHub user maropu opened a pull request:

    https://github.com/apache/spark/pull/22198

    [SPARK-25121][SQL] Supports multi-part table names for broadcast hint resolution

    ## What changes were proposed in this pull request?
    This pr fixed code to respect a database name for broadcast table hint resolution.
    Currently, spark ignores a database name in multi-part names;
    ```
    scala> sql("CREATE DATABASE testDb")
    scala> spark.range(10).write.saveAsTable("testDb.t")
    
    // without this patch
    scala> spark.range(10).join(spark.table("testDb.t"), "id").hint("broadcast", "testDb.t").explain
    == Physical Plan ==
    *(2) Project [id#24L]
    +- *(2) BroadcastHashJoin [id#24L], [id#26L], Inner, BuildLeft
       :- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]))
       :  +- *(1) Range (0, 10, step=1, splits=4)
       +- *(2) Project [id#26L]
          +- *(2) Filter isnotnull(id#26L)
             +- *(2) FileScan parquet testdb.t[id#26L] Batched: true, Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-2.3.1-bin-hadoop2.7/spark-warehouse..., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
    
    // with this patch
    scala> spark.range(10).join(spark.table("testDb.t"), "id").hint("broadcast", "testDb.t").explain
    == Physical Plan ==
    *(2) Project [id#3L]
    +- *(2) BroadcastHashJoin [id#3L], [id#5L], Inner, BuildRight
       :- *(2) Range (0, 10, step=1, splits=4)
       +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true]))
          +- *(1) Project [id#5L]
             +- *(1) Filter isnotnull(id#5L)
                +- *(1) FileScan parquet testdb.t[id#5L] Batched: true, Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-master/spark-warehouse/testdb.db/t], PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
    ```
    
    ## How was this patch tested?
    Added tests in `DataFrameJoinSuite`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/maropu/spark SPARK-25121

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22198.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22198
    
----
commit d2be6920ba1cc052e9d5d8364cf48375cea8ba44
Author: Takeshi Yamamuro <ya...@...>
Date:   2018-08-23T07:20:51Z

    Fix

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by maropu <gi...@git.apache.org>.

Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    **[Test build #95281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95281/testReport)** for PR 22198 at commit [`bd3f502`](https://github.com/apache/spark/commit/bd3f502b573d0e414cece623d7f4bcce52385043).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    @maropu . Did we build the consensus? For me, I'm at the same position until now.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    **[Test build #96505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96505/testReport)** for PR 22198 at commit [`9a6da27`](https://github.com/apache/spark/commit/9a6da2750554cd03ce468d6bfe0aeba8b75883b8).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3422/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22198
  
    **[Test build #95279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95279/testReport)** for PR 22198 at commit [`bd3f502`](https://github.com/apache/spark/commit/bd3f502b573d0e414cece623d7f4bcce52385043).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org