[GitHub] spark pull request #13727: [SPARK-15982][SPARK-16009][SPARK-16007][SQL] Harm...

[marmbrus <gi...@git.apache.org>]

Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13727#discussion_r67572068
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
    @@ -368,6 +397,63 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
        * <li>`maxCharsPerColumn` (default `1000000`): defines the maximum number of characters allowed
        * for any given value being read.</li>
        * <li>`mode` (default `PERMISSIVE`): allows a mode for dealing with corrupt records
    +   * during parsing.</li>
    +   * <ul>
    +   * <li>`PERMISSIVE` : sets other fields to `null` when it meets a corrupted record. When
    +   * a schema is set by user, it sets `null` for extra fields.</li>
    +   * <li>`DROPMALFORMED` : ignores the whole corrupted records.</li>
    +   * <li>`FAILFAST` : throws an exception when it meets corrupted records.</li>
    +   * </ul>
    +   *
    +   * @since 2.0.0
    +   */
    +  def csv(path: String): DataFrame = {
    +    // This method ensures that calls that explicit need single argument works, see SPARK-16009
    +    csv(Seq(path): _*)
    +  }
    +
    +  /**
    +   * Loads a CSV file and returns the result as a [[DataFrame]].
    +   *
    +   * This function will go through the input once to determine the input schema if `inferSchema`
    +   * is enabled. To avoid going through the entire data once, disable `inferSchema` option or
    +   * specify the schema explicitly using [[schema]].
    +   *
    +   * You can set the following CSV-specific options to deal with CSV files:
    +   * <li>`sep` (default `,`): sets the single character as a separator for each
    +   * field and value.</li>
    +   * <li>`encoding` (default `UTF-8`): decodes the CSV files by the given encoding
    +   * type.</li>
    +   * <li>`quote` (default `"`): sets the single character used for escaping quoted values where
    +   * the separator can be part of the value. If you would like to turn off quotations, you need to
    +   * set not `null` but an empty string. This behaviour is different form
    +   * `com.databricks.spark.csv`.</li>
    +   * <li>`escape` (default `\`): sets the single character used for escaping quotes inside
    +   * an already quoted value.</li>
    +   * <li>`comment` (default empty string): sets the single character used for skipping lines
    +   * beginning with this character. By default, it is disabled.</li>
    +   * <li>`header` (default `false`): uses the first line as names of columns.</li>
    +   * <li>`inferSchema` (default `false`): infers the input schema automatically from data. It
    +   * requires one extra pass over the data.</li>
    +   * <li>`ignoreLeadingWhiteSpace` (default `false`): defines whether or not leading whitespaces
    +   * from values being read should be skipped.</li>
    +   * <li>`ignoreTrailingWhiteSpace` (default `false`): defines whether or not trailing
    +   * whitespaces from values being read should be skipped.</li>
    +   * <li>`nullValue` (default empty string): sets the string representation of a null value.</li>
    +   * <li>`nanValue` (default `NaN`): sets the string representation of a non-number" value.</li>
    +   * <li>`positiveInf` (default `Inf`): sets the string representation of a positive infinity
    +   * value.</li>
    +   * <li>`negativeInf` (default `-Inf`): sets the string representation of a negative infinity
    +   * value.</li>
    +   * <li>`dateFormat` (default `null`): sets the string that indicates a date format. Custom date
    +   * formats follow the formats at `java.text.SimpleDateFormat`. This applies to both date type
    +   * and timestamp type. By default, it is `null` which means trying to parse times and date by
    +   * `java.sql.Timestamp.valueOf()` and `java.sql.Date.valueOf()`.</li>
    +   * <li>`maxColumns` (default `20480`): defines a hard limit of how many columns
    +   * a record can have.</li>
    +   * <li>`maxCharsPerColumn` (default `1000000`): defines the maximum number of characters allowed
    +   * for any given value being read.</li>
    +   * <li>`mode` (default `PERMISSIVE`): allows a mode for dealing with corrupt records
    --- End diff --
    
    This can be addressed in a follow up, but I don't think we should duplicate docs cause they are going to get out of sync.  I'd have one canonical one and the other link to it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org