You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by CK50 <gi...@git.apache.org> on 2015/12/15 18:44:27 UTC

[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

GitHub user CK50 opened a pull request:

    https://github.com/apache/spark/pull/10312

    [SPARK-12010][SQL] Add columnMapping support

    In the past Spark JDBC write only worked with technologies which support the following INSERT statement syntax (JdbcUtils.scala: insertStatement()):
    
    INSERT INTO $table VALUES ( ?, ?, ..., ? )
    
    But some technologies require a list of column names:
    
    INSERT INTO $table ( $colNameList ) VALUES ( ?, ?, ..., ? )
    
    This was blocking the use of e.g. the Progress JDBC Driver for Cassandra.
    
    Another limitation is that syntax 1 relies no the dataframe field ordering match that of the target table. This works fine, as long as the target table has been created by writer.jdbc().
    
    If the target table contains more columns (not created by writer.jdbc()), then the insert fails due mismatch of number of columns or their data types.
    
    This PR adds an extra columnMapping parameter to write.jdbc(). This optional parameter allows the user to specify how dataframe field names are mapping to target table column names.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/CK50/spark master-SPARK-12010

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/10312.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #10312
    
----
commit 165b198709e7ef4436e665517fa505b09ba93aab
Author: CK50 <ch...@oracle.com>
Date:   2015-12-15T16:59:06Z

    [SPARK-12010][SQL] Add columnMapping support

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672816
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
    @@ -108,6 +108,34 @@ abstract class JdbcDialect extends Serializable {
       def beforeFetch(connection: Connection, properties: Map[String, String]): Unit = {
       }
     
    +  /**
    +   * Get the SQL statement that should be used to insert new records into the table.
    +   * Dialects can override this method to return a statement that works best in a particular
    +   * database.
    +   * @param table          The name of the table.
    +   * @param rddSchema      The schema of DataFrame to be inserted
    +   * @param columnMapping  An optional mapping from DataFrame field names to database column
    +   *                       names
    +   * @return The SQL statement to use for inserting into the table.
    +   */
    +  def getInsertStatement(table: String,
    +                         rddSchema: StructType,
    +                         columnMapping: Map[String, String] = null): String = {
    +    if (columnMapping == null) {
    +      return rddSchema.fields.map(field => "?")
    --- End diff --
    
    `return` isn't needed. Is this the correct return for this clause? It isn't an insert statement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672218
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -265,10 +266,22 @@ final class DataFrameWriter private[sql](df: DataFrame) {
        * @param connectionProperties JDBC database connection arguments, a list of arbitrary string
        *                             tag/value. Normally at least a "user" and "password" property
        *                             should be included.
    +   * @param columnMapping Maps DataFrame column names to target table column names.
    +   *                           This parameter can be omitted if the target table has/will be
    --- End diff --
    
    Indent on subsequent lines looks 5 spaces too deep. Unless there's a compiler reason it won't work, you can simply make it default to an empty `Map`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47673067
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala ---
    @@ -96,6 +96,16 @@ class JDBCWriteSuite extends SharedSQLContext with BeforeAndAfter {
           2 === sqlContext.read.jdbc(url, "TEST.BASICCREATETEST", new Properties).collect()(0).length)
       }
     
    +  test("Basic CREATE with columnMapping") {
    +    val df = sqlContext.createDataFrame(sparkContext.parallelize(arr2x2), schema2)
    +
    +    val columnMapping = Map("name" -> "name", "id" -> "id")
    +    df.write.jdbc(url, "TEST.BASICCREATETEST", new Properties, columnMapping)
    +    assert(2 === sqlContext.read.jdbc(url, "TEST.BASICCREATETEST", new Properties).count)
    --- End diff --
    
    Final tiny nit -- I'd always put parens on a no-arg constructor call, since it has a side effect by definition. `new Properties()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165519112
  
    Wouldn't a better solution is to just use DataFrame's own renaming of columns? I am not sure why we want this to be a jdbc specific features. If we do this, don't we need to do it for every single other data source?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672592
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala ---
    @@ -61,16 +61,16 @@ object JdbcUtils extends Logging {
     
       /**
        * Returns a PreparedStatement that inserts a row into table via conn.
    +   * If a columnMapping is provided, it will be used to translate rdd
    --- End diff --
    
    nit: rdd -> RDD


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47749421
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
    @@ -108,6 +108,34 @@ abstract class JdbcDialect extends Serializable {
       def beforeFetch(connection: Connection, properties: Map[String, String]): Unit = {
       }
     
    +  /**
    +   * Get the SQL statement that should be used to insert new records into the table.
    +   * Dialects can override this method to return a statement that works best in a particular
    +   * database.
    +   * @param table          The name of the table.
    +   * @param rddSchema      The schema of DataFrame to be inserted
    +   * @param columnMapping  An optional mapping from DataFrame field names to database column
    +   *                       names
    +   * @return The SQL statement to use for inserting into the table.
    +   */
    +  def getInsertStatement(table: String,
    +                         rddSchema: StructType,
    +                         columnMapping: Map[String, String] = null): String = {
    +    if (columnMapping == null) {
    +      return rddSchema.fields.map(field => "?")
    --- End diff --
    
    Yes, the dialect returns the SQL Insert Statement String, which is then turned into a prepared statement in JdbcUtils.insertStatement


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672324
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -265,10 +266,22 @@ final class DataFrameWriter private[sql](df: DataFrame) {
        * @param connectionProperties JDBC database connection arguments, a list of arbitrary string
        *                             tag/value. Normally at least a "user" and "password" property
        *                             should be included.
    +   * @param columnMapping Maps DataFrame column names to target table column names.
    +   *                           This parameter can be omitted if the target table has/will be
    +   *                           created in this method and therefore the target table structure
    +   *                           matches the DF structure.
    +   *                           This parameter is stongly recommended, if target table already
    +   *                           exists and has been created outside of this method.
    +   *                           If omitted, the SQL insert statement will not include column names,
    +   *                           which means that the field ordering of the DataFrame must match
    +   *                           the target table column ordering.
        *
        * @since 1.4.0
        */
    -  def jdbc(url: String, table: String, connectionProperties: Properties): Unit = {
    +  def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    +           columnMapping: scala.collection.immutable.Map[String, String]): Unit = {
    --- End diff --
    
    Any `scala.collection.Map` is OK right? doesn't need to be immutable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47673535
  
    --- Diff: python/pyspark/sql/readwriter.py ---
    @@ -511,13 +511,20 @@ def jdbc(self, url, table, mode=None, properties=None):
             :param properties: JDBC database connection arguments, a list of
                                arbitrary string tag/value. Normally at least a
                                "user" and "password" property should be included.
    +        :param columnMapping: optional column name mapping from DF field names to 
    +                              JDBC table column names.
             """
             if properties is None:
                 properties = dict()
             jprop = JavaClass("java.util.Properties", self._sqlContext._sc._gateway._gateway_client)()
             for k in properties:
                 jprop.setProperty(k, properties[k])
    -        self._jwrite.mode(mode).jdbc(url, table, jprop)
    +        if columnMapping is None:
    +        	  columnMapping = dict()
    --- End diff --
    
    Extra commits are good and fine as it helps show the evolution of the PR. They are squashed in the end. You can squash your own new several commits into one before pushing if it would more logically group and present your subsequent change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672003
  
    --- Diff: python/pyspark/sql/readwriter.py ---
    @@ -511,13 +511,20 @@ def jdbc(self, url, table, mode=None, properties=None):
             :param properties: JDBC database connection arguments, a list of
                                arbitrary string tag/value. Normally at least a
                                "user" and "password" property should be included.
    +        :param columnMapping: optional column name mapping from DF field names to 
    +                              JDBC table column names.
             """
             if properties is None:
                 properties = dict()
             jprop = JavaClass("java.util.Properties", self._sqlContext._sc._gateway._gateway_client)()
             for k in properties:
                 jprop.setProperty(k, properties[k])
    -        self._jwrite.mode(mode).jdbc(url, table, jprop)
    +        if columnMapping is None:
    +        	  columnMapping = dict()
    --- End diff --
    
    Then indent looks too deep here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672570
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +316,32 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-specific) version of jdbc method
    +   */
    +   def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    +           columnMapping: java.util.Map[String, String]): Unit = {
    +    // Convert java Map into immutable scala Map
    +    var sColumnMapping: scala.collection.immutable.Map[String, String] = null
    +    if (columnMapping!=null) {
    +        sColumnMapping = collection.immutable.Map(columnMapping.asScala.toList: _*)
    +    }
    +    jdbc( url, table, connectionProperties, sColumnMapping )
    +  }
    +
    +  /**
    +   * legacy three parameter version of jdbc method
    +   */
    +  def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties): Unit = {
    +    val columnMapping: scala.collection.immutable.Map[String, String] = null
    +    jdbc( url, table, connectionProperties, columnMapping )
    --- End diff --
    
    Also extra spaces here; just pass a `Map()` as the final arg?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47673153
  
    --- Diff: python/pyspark/sql/readwriter.py ---
    @@ -511,13 +511,20 @@ def jdbc(self, url, table, mode=None, properties=None):
             :param properties: JDBC database connection arguments, a list of
                                arbitrary string tag/value. Normally at least a
                                "user" and "password" property should be included.
    +        :param columnMapping: optional column name mapping from DF field names to 
    +                              JDBC table column names.
             """
             if properties is None:
                 properties = dict()
             jprop = JavaClass("java.util.Properties", self._sqlContext._sc._gateway._gateway_client)()
             for k in properties:
                 jprop.setProperty(k, properties[k])
    -        self._jwrite.mode(mode).jdbc(url, table, jprop)
    +        if columnMapping is None:
    +        	  columnMapping = dict()
    --- End diff --
    
    Will do updates. - Can you please remind me of best git commands for 
    updating the PR without creating extra commits?
    
    On 15.12.2015 18:59, Sean Owen wrote:
    >
    > In python/pyspark/sql/readwriter.py 
    > <https://github.com/apache/spark/pull/10312#discussion_r47672003>:
    >
    > >          """
    > >          if properties is None:
    > >              properties = dict()
    > >          jprop = JavaClass("java.util.Properties", self._sqlContext._sc._gateway._gateway_client)()
    > >          for k in properties:
    > >              jprop.setProperty(k, properties[k])
    > > -        self._jwrite.mode(mode).jdbc(url, table, jprop)
    > > +        if columnMapping is None:
    > > +        	  columnMapping = dict()
    >
    > Then indent looks too deep here
    >
    > —
    > Reply to this email directly or view it on GitHub 
    > <https://github.com/apache/spark/pull/10312/files#r47672003>.
    >
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165533757
  
    Explicit naming columns in inserts is ok (good to have actually) - but I wouldn't add a jdbc specific column naming.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47928583
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
    @@ -108,6 +108,33 @@ abstract class JdbcDialect extends Serializable {
       def beforeFetch(connection: Connection, properties: Map[String, String]): Unit = {
       }
     
    +  /**
    +   * Get the SQL statement that should be used to insert new records into the table.
    +   * Dialects can override this method to return a statement that works best in a particular
    +   * database.
    +   * @param table          The name of the table.
    +   * @param rddSchema      The schema of DataFrame to be inserted
    +   * @param columnMapping  An optional mapping from DataFrame field names to database column
    +   *                       names
    +   * @return The SQL statement to use for inserting into the table.
    +   */
    +  def getInsertStatement(table: String,
    +                         rddSchema: StructType,
    +                         columnMapping: scala.collection.Map[String, String] = null): String = {
    +    if (columnMapping == null) {
    +      rddSchema.fields.map(field => "?")
    +      .mkString( s"INSERT INTO $table VALUES (", ", ", " ) ")
    --- End diff --
    
    There's still an extra space before the first arg in both `mkString` calls.
    Also `field => "?"` could be `_ => "?"` for a touch more clarity. Finally I think the `match` statement actually reduces to `columnMapping.getOrElse(field.name, field.name)`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672535
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +316,32 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-specific) version of jdbc method
    +   */
    +   def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    +           columnMapping: java.util.Map[String, String]): Unit = {
    +    // Convert java Map into immutable scala Map
    +    var sColumnMapping: scala.collection.immutable.Map[String, String] = null
    +    if (columnMapping!=null) {
    +        sColumnMapping = collection.immutable.Map(columnMapping.asScala.toList: _*)
    +    }
    +    jdbc( url, table, connectionProperties, sColumnMapping )
    --- End diff --
    
    extra space around args


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165523293
  
    Yeah fair point -- just project / rename a dataframe first. WDYT @CK50 ?
    It doesn't change the underlying point about sometimes needing to explicitly name the columns in the INSERT.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47748967
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +316,32 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-specific) version of jdbc method
    +   */
    +   def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    +           columnMapping: java.util.Map[String, String]): Unit = {
    +    // Convert java Map into immutable scala Map
    +    var sColumnMapping: scala.collection.immutable.Map[String, String] = null
    --- End diff --
    
    I am not sure I got "I agree it does matter whether you support null but you need not."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165720594
  
    Closing and going back to initial approach in new PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165537853
  
    +1 to explicit naming but without the extra jdbc specific ability to rename


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-164839152
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47760673
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -265,10 +266,22 @@ final class DataFrameWriter private[sql](df: DataFrame) {
        * @param connectionProperties JDBC database connection arguments, a list of arbitrary string
        *                             tag/value. Normally at least a "user" and "password" property
        *                             should be included.
    +   * @param columnMapping Maps DataFrame column names to target table column names.
    +   *                           This parameter can be omitted if the target table has/will be
    --- End diff --
    
    It's the `immutable` that is not necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 closed the pull request at:

    https://github.com/apache/spark/pull/10312


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672941
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
    @@ -108,6 +108,34 @@ abstract class JdbcDialect extends Serializable {
       def beforeFetch(connection: Connection, properties: Map[String, String]): Unit = {
       }
     
    +  /**
    +   * Get the SQL statement that should be used to insert new records into the table.
    +   * Dialects can override this method to return a statement that works best in a particular
    +   * database.
    +   * @param table          The name of the table.
    +   * @param rddSchema      The schema of DataFrame to be inserted
    +   * @param columnMapping  An optional mapping from DataFrame field names to database column
    +   *                       names
    +   * @return The SQL statement to use for inserting into the table.
    +   */
    +  def getInsertStatement(table: String,
    +                         rddSchema: StructType,
    +                         columnMapping: Map[String, String] = null): String = {
    +    if (columnMapping == null) {
    +      return rddSchema.fields.map(field => "?")
    +             .mkString( s"INSERT INTO $table VALUES (", ", ", " ) ")
    +    } else {
    +      return rddSchema.fields.map(
    +               field => columnMapping.get(field.name) match {
    +                 case Some(name) => name
    +                 case None => s"<JdbcDialect.getInsertStatement: No entry " +
    --- End diff --
    
    This should just return `field.name`, no?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165543367
  
    @srowen, @rxin, @marmbrus 
    If everyone is fine with *always* using column names in the generated INSERT statement and that these column names will *always* be taken from the DataFrame object, then I can go back to my earlier approach.
    Can you please confirm? If so, I will change code accordingly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672425
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +316,32 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-specific) version of jdbc method
    +   */
    +   def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    --- End diff --
    
    Likewise these are indented 1 space too little


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47928977
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +315,34 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-friendly) version of
    +   * [[DataFrameWriter.jdbc(String,String,Properties,scala.collection.Map[String,String]):]]
    +   */
    +  def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    +           columnMapping: java.util.Map[String, String]): Unit = {
    +    // Convert java Map into scala Map
    +    var sColumnMapping: scala.collection.Map[String, String] = null
    +    if (columnMapping!=null) {
    +        sColumnMapping = columnMapping.asScala
    +    }
    +    jdbc(url, table, connectionProperties, sColumnMapping)
    +  }
    +
    +  /**
    +   * Three parameter version of
    +   * [[DataFrameWriter.jdbc(String,String,Properties,scala.collection.Map[String,String]):]]
    +   */
    +  def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties): Unit = {
    +    val columnMapping: scala.collection.Map[String, String] = null
    +    jdbc(url, table, connectionProperties, columnMapping)
    --- End diff --
    
    Rather than declare a dummy var, I think you can pass `null.asInstanceOf[Map[String, String]]`? or an empty `Map`? I suppose that depends on what the semantics of an empty map should be.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672403
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +316,32 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-specific) version of jdbc method
    --- End diff --
    
    Indent is off by a space; if possible write a scaladoc link to the method above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by 3ourroom <gi...@git.apache.org>.
Github user 3ourroom commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-164841412
  
    
    NAVER - http://www.naver.com/
    --------------------------------------------
    
    3ourroom@naver.com 님께 보내신 메일 <Re: [spark] [SPARK-12010][SQL] Add columnMapping support (#10312)> 이 다음과 같은 이유로 전송 실패했습니다.
    
    --------------------------------------------
    
    받는 사람이 회원님의 메일을 수신차단 하였습니다. 
    
    
    --------------------------------------------



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672127
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -253,6 +253,7 @@ final class DataFrameWriter private[sql](df: DataFrame) {
       }
     
       /**
    +   * (scala-specific)
    --- End diff --
    
    Nit: I don't think this is necessary. It's in the Scala API and technically callable from Java. Call the other one Java-friendly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165506695
  
    @cloud-fan @marmbrus @rxin could we maybe get an opinion on the API change? it's the idea of adding a column mapping to allow for the dataframe's column names to map to different names in a table. I hesitate to commit this without a thumbs up from someone else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-165576548
  
    @CK50 yes I personally think it's safer and more compatible to always put col names in the INSERT statement, and assume the caller has lined up the DataFrame column names. So yes it's your same change but just with the mapping logic backed out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672516
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -303,7 +316,32 @@ final class DataFrameWriter private[sql](df: DataFrame) {
           conn.close()
         }
     
    -    JdbcUtils.saveTable(df, url, table, props)
    +    JdbcUtils.saveTable(df, url, table, props, columnMapping)
    +  }
    +
    +  /**
    +   * (java-specific) version of jdbc method
    +   */
    +   def jdbc(url: String,
    +           table: String,
    +           connectionProperties: Properties,
    +           columnMapping: java.util.Map[String, String]): Unit = {
    +    // Convert java Map into immutable scala Map
    +    var sColumnMapping: scala.collection.immutable.Map[String, String] = null
    --- End diff --
    
    You've already imported the `JavaConverters`, so all of this conversion is just `columnMapping.toScala` I agree it does matter whether you support `null` but you need not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47672664
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala ---
    @@ -234,7 +235,8 @@ object JdbcUtils extends Logging {
           df: DataFrame,
           url: String,
           table: String,
    -      properties: Properties = new Properties()) {
    +      properties: Properties = new Properties(),
    +      columnMapping: Map[String, String] = null) {
    --- End diff --
    
    I have this distant memory that there's something wrong with declaring a Scala method with multiple trailing default args? we'll find out


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by 3ourroom <gi...@git.apache.org>.
Github user 3ourroom commented on the pull request:

    https://github.com/apache/spark/pull/10312#issuecomment-164840588
  
    
    NAVER - http://www.naver.com/
    --------------------------------------------
    
    3ourroom@naver.com 님께 보내신 메일 <[spark] [SPARK-12010][SQL] Add columnMapping support (#10312)> 이 다음과 같은 이유로 전송 실패했습니다.
    
    --------------------------------------------
    
    받는 사람이 회원님의 메일을 수신차단 하였습니다. 
    
    
    --------------------------------------------



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12010][SQL] Add columnMapping support

Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10312#discussion_r47747784
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
    @@ -265,10 +266,22 @@ final class DataFrameWriter private[sql](df: DataFrame) {
        * @param connectionProperties JDBC database connection arguments, a list of arbitrary string
        *                             tag/value. Normally at least a "user" and "password" property
        *                             should be included.
    +   * @param columnMapping Maps DataFrame column names to target table column names.
    +   *                           This parameter can be omitted if the target table has/will be
    --- End diff --
    
    I had followed the example given in line 103:
      def options(options: scala.collection.Map[String, String]): DataFrameWriter = {
    and assumed that this is to emphasize the use of a *Scala* Map instead of a *Java* Map.
    
    Should I replace any use of 
    
    scala.collection.immutable.Map[String, String]
    
    by 
    
    Map[String, String]
    
    ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org