You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/05/05 17:09:18 UTC

[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

GitHub user MaxGekk opened a pull request:

    https://github.com/apache/spark/pull/21247

    [SPARK-24190] Separating JSONOptions for read

    ## What changes were proposed in this pull request?
    
    Currently, restrictions in JSONOptions for `encoding` and `lineSep` are the same for read and for write. For example, a requirement for `lineSep` in the code:
    
    ```
    df.write.option("encoding", "UTF-32BE").json(file)
    ```
    doesn't allow to skip `lineSep` and use its default value `\n` because it throws the exception:
    ```
    equirement failed: The lineSep option must be specified for the UTF-32BE encoding
    java.lang.IllegalArgumentException: requirement failed: The lineSep option must be specified for the UTF-32BE encoding
    ```
    
    In the PR, I propose to separate JSONOptions in read and write, and make JSONOptions in write less restrictive.
    
    ## How was this patch tested?
    
    Added new test for blacklisted encodings in read. And the `lineSep` option was removed in write for some tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/MaxGekk/spark-1 json-options-in-write

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21247
    
----
commit 7ea7dec0a3c5f6694a6a7ef51409a9277aeb733f
Author: Maxim Gekk <ma...@...>
Date:   2018-05-05T16:56:46Z

    Separate JSONOptions in read

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    cc @gengliangwang 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90457/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r186277050
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -137,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient private val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    +      s"""The ${enc} encoding must not be included in the blacklist when multiLine is disabled:
    +         | ${blacklist.mkString(", ")}""".stripMargin)
    +
    +    val isLineSepRequired = !(multiLine == false &&
    +      Charset.forName(enc) != StandardCharsets.UTF_8 && lineSeparator.isEmpty)
    +    require(isLineSepRequired, s"The lineSep option must be specified for the $enc encoding")
    --- End diff --
    
    @MaxGekk, how about we just try to remove this restriction? I thought that's your final goal in 2.4.0.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #91783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91783/testReport)** for PR 21247 at commit [`2285652`](https://github.com/apache/spark/commit/2285652f9df893cf124e98c191ef71cd308e1159).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90444/testReport)** for PR 21247 at commit [`e786953`](https://github.com/apache/spark/commit/e786953f006b747e686abfc1eeeda425ee063a0f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #91783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91783/testReport)** for PR 21247 at commit [`2285652`](https://github.com/apache/spark/commit/2285652f9df893cf124e98c191ef71cd308e1159).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92220/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #92220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92220/testReport)** for PR 21247 at commit [`2ad308d`](https://github.com/apache/spark/commit/2ad308d2c964b377bc6efa607399fb64c00cbebf).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90448/testReport)** for PR 21247 at commit [`9c366a0`](https://github.com/apache/spark/commit/9c366a02a6872adff0e36ce6e22358b40e3004a6).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190][SQL] Allow saving of JSON files in ...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r197503819
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala ---
    @@ -26,7 +26,8 @@ import org.apache.hadoop.mapreduce.{Job, TaskAttemptContext}
     import org.apache.spark.internal.Logging
     import org.apache.spark.sql.{AnalysisException, SparkSession}
     import org.apache.spark.sql.catalyst.InternalRow
    -import org.apache.spark.sql.catalyst.json.{JacksonGenerator, JacksonParser, JSONOptions}
    +import org.apache.spark.sql.catalyst.json.{JacksonGenerator, JacksonParser}
    +import org.apache.spark.sql.catalyst.json.{JSONOptions, JSONOptionsInRead}
    --- End diff --
    
    Just to make Scala's code style checker happy and to fit to the 100 chars restriction. I will check if the style checker is happy if I fold imports to one line.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r195706585
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
    @@ -2427,4 +2427,14 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
           spark.read.option("mode", "PERMISSIVE").option("encoding", "UTF-8").json(Seq(badJson).toDS()),
           Row(badJson))
       }
    +
    +  test("SPARK-24190: restrictions for JSONOptions in read") {
    +    val exception = intercept[IllegalArgumentException] {
    +      spark.read
    +        .option("encoding", "UTF-16")
    +        .json(testFile("test-data/utf16LE.json"))
    --- End diff --
    
    Actually the exception must be thrown for both cases - per-line and multiLine mode. I will check both mode, and both encoding `UTF-16` and `UTF-32`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190][SQL] Allow saving of JSON files in ...

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r197495018
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala ---
    @@ -26,7 +26,8 @@ import org.apache.hadoop.mapreduce.{Job, TaskAttemptContext}
     import org.apache.spark.internal.Logging
     import org.apache.spark.sql.{AnalysisException, SparkSession}
     import org.apache.spark.sql.catalyst.InternalRow
    -import org.apache.spark.sql.catalyst.json.{JacksonGenerator, JacksonParser, JSONOptions}
    +import org.apache.spark.sql.catalyst.json.{JacksonGenerator, JacksonParser}
    +import org.apache.spark.sql.catalyst.json.{JSONOptions, JSONOptionsInRead}
    --- End diff --
    
    question/nit: Why is this split on 2 import statements this way?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91344/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #91344 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91344/testReport)** for PR 21247 at commit [`2285652`](https://github.com/apache/spark/commit/2285652f9df893cf124e98c191ef71cd308e1159).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90448/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90257/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Jenkins ok to test.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r194147472
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    @HyukjinKwon I have already implemented `lineSep` detection for different encodings (for `UTF-16` and `UTF-32` in particular). At the moment I am writing tests for that. I will prepare a PR soon.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r191831524
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
    @@ -2313,25 +2314,6 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
         }
       }
     
    -  test("SPARK-23723: write json in UTF-16/32 with multiline off") {
    -    Seq("UTF-16", "UTF-32").foreach { encoding =>
    -      withTempPath { path =>
    -        val ds = spark.createDataset(Seq(
    -          ("a", 1), ("b", 2), ("c", 3))
    -        ).repartition(2)
    -        val e = intercept[IllegalArgumentException] {
    -          ds.write
    -            .option("encoding", encoding)
    -            .option("multiline", "false")
    -            .format("json").mode("overwrite")
    -            .save(path.getCanonicalPath)
    -        }.getMessage
    -        assert(e.contains(
    -          s"$encoding encoding in the blacklist is not allowed when multiLine is disabled"))
    --- End diff --
    
    We can still keep this test case, right? We can change this negative test case to positive test case


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #92183 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92183/testReport)** for PR 21247 at commit [`ca1b243`](https://github.com/apache/spark/commit/ca1b24322edd119d1e15b39f79bb15dd22cae482).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class ForeachBatchFunction(object):`
      * `case class ArrayDistinct(child: Expression)`
      * `class PythonForeachWriter(func: PythonFunction, schema: StructType)`
      * `  class UnsafeRowBuffer(taskMemoryManager: TaskMemoryManager, tempDir: File, numFields: Int)`
      * `trait MemorySinkBase extends BaseStreamingSink with Logging `
      * `class MemorySink(val schema: StructType, outputMode: OutputMode, options: DataSourceOptions)`
      * `class ForeachBatchSink[T](batchWriter: (Dataset[T], Long) => Unit, encoder: ExpressionEncoder[T])`
      * `trait PythonForeachBatchFunction `
      * `case class ForeachWriterProvider[T](`
      * `case class ForeachWriterFactory[T](`
      * `class ForeachDataWriter[T](`
      * `class MemoryWriter(`
      * `class MemoryStreamWriter(`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r187780271
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    There is no reasons to blacklist `UTF-16` and `UTF-32` in write. I have checked the content of written JSON files on @gatorsmile 's [test](https://github.com/apache/spark/pull/21247/commits/97c4af76addc78a85ceb503a5db16f3285f18a5f). For example, for `UTF-16`
    ```
    $ hexdump -C ...c000.json
    00000000  fe ff 00 7b 00 22 00 5f  00 31 00 22 00 3a 00 22  |...{."._.1.".:."|
    00000010  00 61 00 22 00 2c 00 22  00 5f 00 32 00 22 00 3a  |.a.".,."._.2.".:|
    00000020  00 31 00 7d 00 0a 00 7b  00 22 00 5f 00 31 00 22  |.1.}...{."._.1."|
    00000030  00 3a 00 22 00 63 00 22  00 2c 00 22 00 5f 00 32  |.:.".c.".,."._.2|
    00000040  00 22 00 3a 00 33 00 7d  00 0a                    |.".:.3.}..|
    0000004a
    ```
    It contains BOM `fe ff` at the beginning as it is expected, and written line separator doesn't contains BOM (look at the position 0x24-0x25) - `00 7d` **00 0a** `00 7b`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #92220 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92220/testReport)** for PR 21247 at commit [`2ad308d`](https://github.com/apache/spark/commit/2ad308d2c964b377bc6efa607399fb64c00cbebf).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r194117346
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    So I could be missing something, but it seems like we might allow folks to write data they can't read back in the same mode as write, would it make sense to have an equivalent checkedEncoding on the write side that just logs a warning for folks? I could have also misunderstood.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by crafty-coder <gi...@git.apache.org>.
Github user crafty-coder commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Hi @holdenk , I'm working on a similar PR (https://github.com/apache/spark/pull/20949) to allow setting up the encoding when writing csv files.
    
    It would be strange if you could set up the encoding when saving a json file and you can't do the same when saving a csv.
    
    It would be nice if you could take a look! 
    
    Thanks you 🐱 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #91344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91344/testReport)** for PR 21247 at commit [`2285652`](https://github.com/apache/spark/commit/2285652f9df893cf124e98c191ef71cd308e1159).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r186277493
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -137,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient private val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    BTW, don't we still need the blacklisting in write site too?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90257/testReport)** for PR 21247 at commit [`7ea7dec`](https://github.com/apache/spark/commit/7ea7dec0a3c5f6694a6a7ef51409a9277aeb733f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r195151361
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -94,32 +94,15 @@ private[sql] class JSONOptions(
         sep
       }
     
    +  protected def checkedEncoding(enc: String): String = enc
    +
       /**
        * Standard encoding (charset) name. For example UTF-8, UTF-16LE and UTF-32BE.
    -   * If the encoding is not specified (None), it will be detected automatically
    +   * If the encoding is not specified (None) in read, it will be detected automatically
    --- End diff --
    
    Since this comment mentions what happens in the read path, it would probably be good to mention what happens in the write path (e.g. based on `JsonFileFormat.scala` it looks like in the write path it defaults to UTF-8.



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190][SQL] Allow saving of JSON files in ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21247


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r186283555
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -137,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient private val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    +      s"""The ${enc} encoding must not be included in the blacklist when multiLine is disabled:
    +         | ${blacklist.mkString(", ")}""".stripMargin)
    +
    +    val isLineSepRequired = !(multiLine == false &&
    +      Charset.forName(enc) != StandardCharsets.UTF_8 && lineSeparator.isEmpty)
    +    require(isLineSepRequired, s"The lineSep option must be specified for the $enc encoding")
    --- End diff --
    
    Do you mean rewriting Hadoop's LineReader to detect `\n`, `\r` and `\r\n` for any encoding? If so, I am working on it but I think we shouldn't restrict writer till we remove the restriction for reader.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r194123992
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    Yup and I think the final goal within 2.4.0 is to get rid of the blacklists in both read and writr. Shall we just focus on getting rid of it?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90457/testReport)** for PR 21247 at commit [`97c4af7`](https://github.com/apache/spark/commit/97c4af76addc78a85ceb503a5db16f3285f18a5f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92183/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r195145029
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    So in that case logging the warning would be less important if we can read them back. In-case we can't would having the warning for now be ok?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90457/testReport)** for PR 21247 at commit [`97c4af7`](https://github.com/apache/spark/commit/97c4af76addc78a85ceb503a5db16f3285f18a5f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r194119938
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
    @@ -2427,4 +2427,14 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
           spark.read.option("mode", "PERMISSIVE").option("encoding", "UTF-8").json(Seq(badJson).toDS()),
           Row(badJson))
       }
    +
    +  test("SPARK-24190: restrictions for JSONOptions in read") {
    +    val exception = intercept[IllegalArgumentException] {
    +      spark.read
    +        .option("encoding", "UTF-16")
    +        .json(testFile("test-data/utf16LE.json"))
    --- End diff --
    
    super minor, but to more closely match the previous test maybe explicitly set multi-line to false explicitly.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90448/testReport)** for PR 21247 at commit [`9c366a0`](https://github.com/apache/spark/commit/9c366a02a6872adff0e36ce6e22358b40e3004a6).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90444/testReport)** for PR 21247 at commit [`e786953`](https://github.com/apache/spark/commit/e786953f006b747e686abfc1eeeda425ee063a0f).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91783/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r187769734
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    Do we need to check `blacklist` in write path?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #91916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91916/testReport)** for PR 21247 at commit [`c1971a5`](https://github.com/apache/spark/commit/c1971a5214d38c634e93b53e63ecbefeac8aba6e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r195713474
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -138,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient override val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    --- End diff --
    
    The written JSON in the blacklisted encodings can be read back if `multiLine` is enabled. We can output a warning with such hint.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #92183 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92183/testReport)** for PR 21247 at commit [`ca1b243`](https://github.com/apache/spark/commit/ca1b24322edd119d1e15b39f79bb15dd22cae482).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r192106493
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
    @@ -2313,25 +2314,6 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
         }
       }
     
    -  test("SPARK-23723: write json in UTF-16/32 with multiline off") {
    -    Seq("UTF-16", "UTF-32").foreach { encoding =>
    -      withTempPath { path =>
    -        val ds = spark.createDataset(Seq(
    -          ("a", 1), ("b", 2), ("c", 3))
    -        ).repartition(2)
    -        val e = intercept[IllegalArgumentException] {
    -          ds.write
    -            .option("encoding", encoding)
    -            .option("multiline", "false")
    -            .format("json").mode("overwrite")
    -            .save(path.getCanonicalPath)
    -        }.getMessage
    -        assert(e.contains(
    -          s"$encoding encoding in the blacklist is not allowed when multiLine is disabled"))
    --- End diff --
    
    Yes, sure. I converted the test to positive one.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21247#discussion_r186284065
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -137,3 +121,40 @@ private[sql] class JSONOptions(
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
     }
    +
    +private[sql] class JSONOptionsInRead(
    +    @transient private val parameters: CaseInsensitiveMap[String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String)
    +  extends JSONOptions(parameters, defaultTimeZoneId, defaultColumnNameOfCorruptRecord) {
    +
    +  def this(
    +    parameters: Map[String, String],
    +    defaultTimeZoneId: String,
    +    defaultColumnNameOfCorruptRecord: String = "") = {
    +    this(
    +      CaseInsensitiveMap(parameters),
    +      defaultTimeZoneId,
    +      defaultColumnNameOfCorruptRecord)
    +  }
    +
    +  protected override def checkedEncoding(enc: String): String = {
    +    // The following encodings are not supported in per-line mode (multiline is false)
    +    // because they cause some problems in reading files with BOM which is supposed to
    +    // present in the files with such encodings. After splitting input files by lines,
    +    // only the first lines will have the BOM which leads to impossibility for reading
    +    // the rest lines. Besides of that, the lineSep option must have the BOM in such
    +    // encodings which can never present between lines.
    +    val blacklist = Seq(Charset.forName("UTF-16"), Charset.forName("UTF-32"))
    +    val isBlacklisted = blacklist.contains(Charset.forName(enc))
    +    require(multiLine || !isBlacklisted,
    +      s"""The ${enc} encoding must not be included in the blacklist when multiLine is disabled:
    +         | ${blacklist.mkString(", ")}""".stripMargin)
    +
    +    val isLineSepRequired = !(multiLine == false &&
    +      Charset.forName(enc) != StandardCharsets.UTF_8 && lineSeparator.isEmpty)
    +    require(isLineSepRequired, s"The lineSep option must be specified for the $enc encoding")
    --- End diff --
    
    Yea, and also I thought you are working on getting rid of blacklisting encodings too. Roughly this PR makes sense for the intermediate status since we have different requirements in write and read path; however, I thought we should just better try to remove the restrictions first until the release becomes close, and the current change should be done in the last minute if we failed to get rid of the restrictions.
    
    I am a bit cautious of the current change since it's pretty new approach to datasources. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    LGTM
    
    Thanks! Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #90257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90257/testReport)** for PR 21247 at commit [`7ea7dec`](https://github.com/apache/spark/commit/7ea7dec0a3c5f6694a6a7ef51409a9277aeb733f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91916/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90444/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    **[Test build #91916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91916/testReport)** for PR 21247 at commit [`c1971a5`](https://github.com/apache/spark/commit/c1971a5214d38c634e93b53e63ecbefeac8aba6e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21247: [SPARK-24190] Separating JSONOptions for read

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21247
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org