You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2018/05/21 03:38:54 UTC

[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

GitHub user maropu opened a pull request:

    https://github.com/apache/spark/pull/21379

    [SPARK-24327][SQL] Add an option to quote a partition column name in JDBCRelation

    ## What changes were proposed in this pull request?
    This pr added a new option to quote a partition column name in `JDBCRelation`.
    
    ## How was this patch tested?
    Added tests in `JDBCSuite`.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/maropu/spark SPARK-24327

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21379.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21379
    
----
commit 8d97b0deb5ed96094f70f16376b677fe3ff1bdfc
Author: Takeshi Yamamuro <ya...@...>
Date:   2018-05-21T03:36:16Z

    Fix

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r191117129
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +79,12 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +    val column = if (jdbcOptions.quotePartitionColumnName) {
    +      val dialect = JdbcDialects.get(jdbcOptions.url)
    +      dialect.quoteIdentifier(partitioning.column)
    --- End diff --
    
    [The latest fix](https://github.com/apache/spark/pull/21379/commits/8d5fa9a25dff75327e8ff8ff13b756b895e6e512) changes an existing behaviour (when quoting non-partition column names), so I'm not sure this fix is acceptable. Any suggestion?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    LGTM except one minor comment.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91813/testReport)** for PR 21379 at commit [`d76bc7f`](https://github.com/apache/spark/commit/d76bc7fc555bbfe4da25c959646a6ee5961d4d14).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/121/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    You mean we need verification code there for checking if the fetched schema has a user-given partition column? If the schema does not have the column, throws `AnalysisException` or something?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3401/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    @gatorsmile @conorbmurphy


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91814/testReport)** for PR 21379 at commit [`a3be215`](https://github.com/apache/spark/commit/a3be215755f00100be0817b2a59f1ea8a185518b).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91814/testReport)** for PR 21379 at commit [`a3be215`](https://github.com/apache/spark/commit/a3be215755f00100be0817b2a59f1ea8a185518b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91812/testReport)** for PR 21379 at commit [`a9b0306`](https://github.com/apache/spark/commit/a9b030682be358f36c0d2e64b175017458774b20).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4009/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3630/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4026/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91812/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #90882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90882/testReport)** for PR 21379 at commit [`8d97b0d`](https://github.com/apache/spark/commit/8d97b0deb5ed96094f70f16376b677fe3ff1bdfc).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r189718595
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +79,12 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +    val column = if (jdbcOptions.quotePartitionColumnName) {
    +      val dialect = JdbcDialects.get(jdbcOptions.url)
    +      dialect.quoteIdentifier(partitioning.column)
    --- End diff --
    
    If the input `partition.column` is already quoted, can we skip the quotes?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92285/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/135/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Verify and normalize a partiti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195479735
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
    @@ -309,7 +309,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
         val parts: Array[Partition] = predicates.zipWithIndex.map { case (part, i) =>
           JDBCPartition(part, i) : Partition
         }
    -    val relation = JDBCRelation(parts, options)(sparkSession)
    +    val schema = JDBCRelation.getSchema(sparkSession.sessionState.conf.resolver, options)
    +    val relation = JDBCRelation(schema, parts, options)(sparkSession)
    --- End diff --
    
    We do not need to change this. Add an apply function to object JDBCRelation.scala
    ```
      def apply(parts: Array[Partition], jdbcOptions: JDBCOptions)(
          sparkSession: SparkSession): JDBCRelation = {
        val schema = getSchema(jdbcOptions, sparkSession.sessionState.conf.resolver)
        JDBCRelation(schema, parts, jdbcOptions)(sparkSession)
      }
    ```
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4010/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #90874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90874/testReport)** for PR 21379 at commit [`8d97b0d`](https://github.com/apache/spark/commit/8d97b0deb5ed96094f70f16376b677fe3ff1bdfc).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195313934
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -51,7 +53,11 @@ private[sql] object JDBCRelation extends Logging {
        * @param partitioning partition information to generate the where clause for each partition
    --- End diff --
    
    Add the other two `@param` for the new parameters?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91867/testReport)** for PR 21379 at commit [`614c87a`](https://github.com/apache/spark/commit/614c87aada943c16faa3f7a718837de0e4dcaeab).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91867/testReport)** for PR 21379 at commit [`614c87a`](https://github.com/apache/spark/commit/614c87aada943c16faa3f7a718837de0e4dcaeab).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    ok, I will re-check.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4045/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/452/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Sorry, I just realized this is a wrong direction. Instead of trusting the user inputs, we should verify the user-specified partition columns by using the already fetched table schema info `val tableSchema = JDBCRDD.resolveTable(jdbcOptions)` when building `JDBCRelation`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195314034
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +84,17 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +
    +    // Verify and normalize a partition column based on the JDBC resolved schema
    +    val dialect = JdbcDialects.get(jdbcOptions.url)
    +    val column = schema.map(_.name).find { fieldName =>
    +      resolver(fieldName, partitioning.column) ||
    +        resolver(dialect.quoteIdentifier(fieldName), partitioning.column)
    +    }.map(dialect.quoteIdentifier).getOrElse {
    +      throw new AnalysisException(s"User-defined partition column ${partitioning.column} not " +
    +        s"found in the JDBC relation: ${schema.simpleString(Utils.maxNumToStringFields)}")
    +    }
    --- End diff --
    
    Create a new private function for the above resolution and checking logic?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/119/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #90874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90874/testReport)** for PR 21379 at commit [`8d97b0d`](https://github.com/apache/spark/commit/8d97b0deb5ed96094f70f16376b677fe3ff1bdfc).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/120/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91213/testReport)** for PR 21379 at commit [`8d5fa9a`](https://github.com/apache/spark/commit/8d5fa9a25dff75327e8ff8ff13b756b895e6e512).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/137/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r190450621
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +79,12 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +    val column = if (jdbcOptions.quotePartitionColumnName) {
    +      val dialect = JdbcDialects.get(jdbcOptions.url)
    +      dialect.quoteIdentifier(partitioning.column)
    --- End diff --
    
    ok, I will


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Yeah, we should do it. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91814/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91813/testReport)** for PR 21379 at commit [`d76bc7f`](https://github.com/apache/spark/commit/d76bc7fc555bbfe4da25c959646a6ee5961d4d14).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91835/testReport)** for PR 21379 at commit [`28e811e`](https://github.com/apache/spark/commit/28e811e2703d84a70f331a284d9d8a3ac01ec75a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #92285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92285/testReport)** for PR 21379 at commit [`614c87a`](https://github.com/apache/spark/commit/614c87aada943c16faa3f7a718837de0e4dcaeab).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91835/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    oh, I forgot to update the title..yea, I'll do soon.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3408/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    The build passed. The tests have passed in the previous run. The current tests will be killed at the midnight. 
    
    LGTM
    
    Thanks! Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91838/testReport)** for PR 21379 at commit [`d2ef95c`](https://github.com/apache/spark/commit/d2ef95c9c1ddb446a1ffda3935ea783e6e17b114).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Verify and normalize a partiti...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195574001
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
    @@ -309,7 +309,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
         val parts: Array[Partition] = predicates.zipWithIndex.map { case (part, i) =>
           JDBCPartition(part, i) : Partition
         }
    -    val relation = JDBCRelation(parts, options)(sparkSession)
    +    val schema = JDBCRelation.getSchema(sparkSession.sessionState.conf.resolver, options)
    +    val relation = JDBCRelation(schema, parts, options)(sparkSession)
    --- End diff --
    
    ok


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90882/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/154/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #92285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92285/testReport)** for PR 21379 at commit [`614c87a`](https://github.com/apache/spark/commit/614c87aada943c16faa3f7a718837de0e4dcaeab).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Verify and normalize a partiti...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21379


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91213/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91838 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91838/testReport)** for PR 21379 at commit [`d2ef95c`](https://github.com/apache/spark/commit/d2ef95c9c1ddb446a1ffda3935ea783e6e17b114).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r190382338
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +79,12 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +    val column = if (jdbcOptions.quotePartitionColumnName) {
    +      val dialect = JdbcDialects.get(jdbcOptions.url)
    +      dialect.quoteIdentifier(partitioning.column)
    --- End diff --
    
    Yeah


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195315740
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -51,7 +53,11 @@ private[sql] object JDBCRelation extends Logging {
        * @param partitioning partition information to generate the where clause for each partition
    --- End diff --
    
    ok


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91835/testReport)** for PR 21379 at commit [`28e811e`](https://github.com/apache/spark/commit/28e811e2703d84a70f331a284d9d8a3ac01ec75a).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4024/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    ping


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91813/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91867/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195313861
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -99,10 +115,21 @@ private[sql] object JDBCRelation extends Logging {
         }
         ans.toArray
       }
    +
    +  def getSchema(jdbcOptions: JDBCOptions, resolver: Resolver): StructType = {
    --- End diff --
    
    Add the function description? 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r189774838
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +79,12 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +    val column = if (jdbcOptions.quotePartitionColumnName) {
    +      val dialect = JdbcDialects.get(jdbcOptions.url)
    +      dialect.quoteIdentifier(partitioning.column)
    --- End diff --
    
    You point out the case users explicitly add quotes in `partitionColumn`? e.g.,
    ```
    +      val df = spark.read.format("jdbc")
    +        ...
    +        .option("partitionColumn", """"THEID"""")
    +        ...
    +        .option("quotePartitionColumnName", quotePartitionColumnName)
    +        .load()
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4008/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91838/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91213/testReport)** for PR 21379 at commit [`8d5fa9a`](https://github.com/apache/spark/commit/8d5fa9a25dff75327e8ff8ff13b756b895e6e512).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r191641304
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala ---
    @@ -78,7 +79,12 @@ private[sql] object JDBCRelation extends Logging {
         // Overflow and silliness can happen if you subtract then divide.
         // Here we get a little roundoff, but that's (hopefully) OK.
         val stride: Long = upperBound / numPartitions - lowerBound / numPartitions
    -    val column = partitioning.column
    +    val column = if (jdbcOptions.quotePartitionColumnName) {
    +      val dialect = JdbcDialects.get(jdbcOptions.url)
    +      dialect.quoteIdentifier(partitioning.column)
    --- End diff --
    
    If possible, please do not break the existing behavior.  Both should work without specifying the extra option:
    ```
    val df = spark.read.format("jdbc")
      .option("url", urlWithUserAndPass)
      .option("dbtable", "TEST.PEOPLE")
      .option("partitionColumn", "nonQuotedPrtColName")
      .option("lowerBound", 1)
      .option("upperBound", 4)
      .option("numPartitions", 3)
      .load()
    ```
    ```
    val df = spark.read.format("jdbc")
      .option("url", urlWithUserAndPass)
      .option("dbtable", "TEST.PEOPLE")
      .option("partitionColumn", """"nonQuotedPrtColName"""")
      .option("lowerBound", 1)
      .option("upperBound", 4)
      .option("numPartitions", 3)
      .load()
    ```
    
    Is that possible?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #90882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90882/testReport)** for PR 21379 at commit [`8d97b0d`](https://github.com/apache/spark/commit/8d97b0deb5ed96094f70f16376b677fe3ff1bdfc).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90874/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    **[Test build #91812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91812/testReport)** for PR 21379 at commit [`a9b0306`](https://github.com/apache/spark/commit/a9b030682be358f36c0d2e64b175017458774b20).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    @maropu The fix looks good to me. Thanks for changing the solution. Could you update the PR title and description?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21379
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21379#discussion_r195313083
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala ---
    @@ -1206,4 +1211,47 @@ class JDBCSuite extends SparkFunSuite
         }.getMessage
         assert(errMsg.contains("Statement was canceled or the session timed out"))
       }
    +
    +  test("SPARK-24327 verify and normalize a partition column based on a JDBC resolved schema") {
    +    def testJdbcParitionColumn(partColName: String, expectedColumnName: String): Unit = {
    +      val df = spark.read.format("jdbc")
    +        .option("url", urlWithUserAndPass)
    +        .option("dbtable", "TEST.PARTITION")
    +        .option("partitionColumn", partColName)
    +        .option("lowerBound", 1)
    +        .option("upperBound", 4)
    +        .option("numPartitions", 3)
    +        .load()
    +
    +      val quotedPrtColName = testH2Dialect.quoteIdentifier(expectedColumnName)
    +      df.logicalPlan match {
    +        case LogicalRelation(JDBCRelation(_, parts, _), _, _, _) =>
    +          val whereClauses = parts.map(_.asInstanceOf[JDBCPartition].whereClause).toSet
    +          assert(whereClauses === Set(
    +            s"$quotedPrtColName < 2 or $quotedPrtColName is null",
    +            s"$quotedPrtColName >= 2 AND $quotedPrtColName < 3",
    +            s"$quotedPrtColName >= 3"))
    +      }
    +    }
    +
    +    testJdbcParitionColumn("THEID", "THEID")
    +    testJdbcParitionColumn("\"THEID\"", "THEID")
    +    withSQLConf("spark.sql.caseSensitive" -> "false") {
    +      testJdbcParitionColumn("ThEiD", "THEID")
    +    }
    +    testJdbcParitionColumn("THE ID", "THE ID")
    +
    +    def testIncorrectJdbcPartitionColumn(partColName: String): Unit = {
    +      val errMsg = intercept[AnalysisException] {
    +        testJdbcParitionColumn(partColName, "THEID")
    +      }.getMessage
    +      assert(errMsg.contains(s"User-defined partition column $partColName not found " +
    +        "in the JDBC relation:"))
    +    }
    +
    +    testIncorrectJdbcPartitionColumn("NoExistingColumn")
    +    withSQLConf("spark.sql.caseSensitive" -> "true") {
    --- End diff --
    
    `SQLConf.CASE_SENSITIVE.key`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org