You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/09/06 11:33:14 UTC

[GitHub] spark pull request #22351: [MINOR][SQL] Add a debug log when a SQL text is u...

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/22351

    [MINOR][SQL] Add a debug log when a SQL text is used for a view

    ## What changes were proposed in this pull request?
    
    This took me a while to debug and find out. Looks we better at least leave a debug log that SQL text for a view will be used.
    
    Here's how I got there:
    
    **Hive:**
    
    ```
    CREATE TABLE emp AS SELECT 'user' AS name, 'address' as address;
    CREATE DATABASE d100;
    CREATE FUNCTION d100.udf100 AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper';
    CREATE VIEW testview AS SELECT d100.udf100(name) FROM default.emp;
    ```
    
    **Spark:**
    
    ```
    sql("SELECT * FROM testview").show()
    ```
    
    ```
    scala> sql("SELECT * FROM testview").show()
    org.apache.spark.sql.AnalysisException: Undefined function: 'd100.udf100'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7
    ```
    
    
    Under the hood, it actually makes sense since the view is defined as `SELECT d100.udf100(name) FROM default.emp;` and Hive API:
    
    ```
    org.apache.hadoop.hive.ql.metadata.Table.getViewExpandedText()
    ```
    
    This returns a fully qualified SQL string for the view as below:
    
    ```
    SELECT `d100.udf100`(`emp`.`name`) FROM `default`.`emp`
    ```
    
    which works fine in Hive but not in Spark.
    
    
    ## How was this patch tested?
    
    Manually:
    
    ```
    18/09/06 19:32:48 DEBUG HiveSessionCatalog: 'SELECT `d100.udf100`(`emp`.`name`) FROM `default`.`emp`' will be used for the view(testview).
    ```


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark minor-debug

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22351
    
----
commit 207d8dfb240ce8e283716e66bb62e7330a69fa8f
Author: hyukjinkwon <gu...@...>
Date:   2018-09-06T11:30:10Z

    Add a debug log when a SQL text is used for a view

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95754/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    cc @cloud-fan


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by jiangxb1987 <gi...@git.apache.org>.

Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    Just confirmed if the view is created and retrieved both at Spark side then there will be no exception thrown.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by jiangxb1987 <gi...@git.apache.org>.

Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    This is actually read some view created by Hive, so I don't think it shall be a problem with view write side.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2899/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by cloud-fan <gi...@git.apache.org>.

Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    I'm surprised Hive changes the view text set by Spark. Is it a problem for views? cc @gatorsmile @jiangxb1987 @hvanhovell 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    **[Test build #95754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95754/testReport)** for PR 22351 at commit [`207d8df`](https://github.com/apache/spark/commit/207d8dfb240ce8e283716e66bb62e7330a69fa8f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    **[Test build #95754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95754/testReport)** for PR 22351 at commit [`207d8df`](https://github.com/apache/spark/commit/207d8dfb240ce8e283716e66bb62e7330a69fa8f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22351: [MINOR][SQL] Add a debug log when a SQL text is used for...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22351
  
    Done, thanks @dongjoon-hyun 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org