You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/08/24 19:29:05 UTC

[GitHub] spark pull request #22226: [SPARK-24391][SQL] Support arrays of any types by...

GitHub user MaxGekk opened a pull request:

    https://github.com/apache/spark/pull/22226

    [SPARK-24391][SQL] Support arrays of any types by to_json

    ## What changes were proposed in this pull request?
    
    In the PR, I propose to extended `to_json` and support any types as element types of input arrays. It should allow converting arrays of primitive types and arrays of arrays. For example:
    
    ```
    select to_json(array('1','2','3'))
    > ["1","2","3"]
    select to_json(array(array(1,2,3),array(4)))
    > [[1,2,3],[4]]
    ``` 
    
    ## How was this patch tested?
    
    Added a couple sql tests for arrays of primitive type and of arrays. Also I added round trip test `from_json` -> `to_json`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/MaxGekk/spark-1 to_json-array

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22226.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22226
    
----
commit fb727268b776f3c9b896abfee60e8538aa9e93e4
Author: Maxim Gekk <ma...@...>
Date:   2018-08-24T18:57:36Z

    Support arrays of any types in to_json

commit e9a5f50720d0e6ab301afaa3a4eff801de1240d7
Author: Maxim Gekk <ma...@...>
Date:   2018-08-24T19:18:19Z

    sql tests for array of primitive type and of arrays

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95291/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95227/testReport)** for PR 22226 at commit [`e9a5f50`](https://github.com/apache/spark/commit/e9a5f50720d0e6ab301afaa3a4eff801de1240d7).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214157102
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
    @@ -3493,11 +3493,11 @@ object functions {
       def schema_of_json(e: Column): Column = withExpr(new SchemaOfJson(e.expr))
     
       /**
    -   * (Scala-specific) Converts a column containing a `StructType`, `ArrayType` of `StructType`s,
    -   * a `MapType` or `ArrayType` of `MapType`s into a JSON string with the specified schema.
    +   * (Scala-specific) Converts a column containing a `StructType`, `ArrayType` or
    +   * a `MapType` into a JSON string with the specified schema.
        * Throws an exception, in the case of an unsupported type.
        *
    -   * @param e a column containing a struct or array of the structs.
    +   * @param e a column containing a struct, a array or a map.
    --- End diff --
    
    typo: `a array` -> `an array`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    @maropu The JIRA ticket was about both `to_json` and `from_json` originally.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95235/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    +1 for separate JIRA.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214843459
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -685,33 +679,29 @@ case class StructsToJson(
             (row: Any) =>
               gen.write(row.asInstanceOf[InternalRow])
               getAndReset()
    -      case ArrayType(_: StructType, _) =>
    +      case _: ArrayType =>
             (arr: Any) =>
               gen.write(arr.asInstanceOf[ArrayData])
               getAndReset()
           case _: MapType =>
             (map: Any) =>
               gen.write(map.asInstanceOf[MapData])
               getAndReset()
    -      case ArrayType(_: MapType, _) =>
    -        (arr: Any) =>
    -          gen.write(arr.asInstanceOf[ArrayData])
    -          getAndReset()
         }
       }
     
       override def dataType: DataType = StringType
     
       override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
    --- End diff --
    
    nit: `child.dataType` -> `inputSchema`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214878373
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -663,12 +662,7 @@ case class StructsToJson(
         rowSchema, writer, new JSONOptions(options, timeZoneId.get))
     
       @transient
    -  lazy val rowSchema = child.dataType match {
    -    case st: StructType => st
    -    case ArrayType(st: StructType, _) => st
    -    case mt: MapType => mt
    -    case ArrayType(mt: MapType, _) => mt
    -  }
    +  lazy val rowSchema = child.dataType
    --- End diff --
    
    I tried to remove `lazy` and got many errors on tests like:
    ```
    Invalid call to dataType on unresolved object, tree: 'a
    org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'a
    	at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:105)
    	at org.apache.spark.sql.catalyst.expressions.StructsToJson.<init>(jsonExpressions.scala:665)
    ```
    If you don't mind, I will keep it `lazy`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95622/testReport)** for PR 22226 at commit [`72d2628`](https://github.com/apache/spark/commit/72d2628323af4e44da1083c99c0d4996c34e4c8c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214844726
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -663,12 +662,7 @@ case class StructsToJson(
         rowSchema, writer, new JSONOptions(options, timeZoneId.get))
     
       @transient
    -  lazy val rowSchema = child.dataType match {
    -    case st: StructType => st
    -    case ArrayType(st: StructType, _) => st
    -    case mt: MapType => mt
    -    case ArrayType(mt: MapType, _) => mt
    -  }
    +  lazy val rowSchema = child.dataType
    --- End diff --
    
    Let's make it `val`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95671/testReport)** for PR 22226 at commit [`90c9687`](https://github.com/apache/spark/commit/90c968772d74c8aadc7a4d0e74e226554b921486).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r213177790
  
    --- Diff: python/pyspark/sql/functions.py ---
    @@ -2289,12 +2289,10 @@ def from_json(col, schema, options={}):
     @since(2.1)
     def to_json(col, options={}):
         """
    -    Converts a column containing a :class:`StructType`, :class:`ArrayType` of
    -    :class:`StructType`\\s, a :class:`MapType` or :class:`ArrayType` of :class:`MapType`\\s
    +    Converts a column containing a :class:`StructType`, :class:`ArrayType` or a :class:`MapType`
         into a JSON string. Throws an exception, in the case of an unsupported type.
     
    -    :param col: name of column containing the struct, array of the structs, the map or
    -        array of the maps.
    +    :param col: name of column containing a struct, an array or a map.
         :param options: options to control converting. accepts the same options as the json datasource
    --- End diff --
    
    ditto


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214156762
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
    @@ -43,20 +42,22 @@ private[sql] class JacksonGenerator(
       // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
       private type ValueWriter = (SpecializedGetters, Int) => Unit
     
    -  // `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
    -  require(dataType.isInstanceOf[StructType] || dataType.isInstanceOf[MapType],
    +  // `JackGenerator` can only be initialized with a `StructType`, a `MapType` or a `ArrayType`.
    +  require(dataType.isInstanceOf[StructType] || dataType.isInstanceOf[MapType]
    +    || dataType.isInstanceOf[ArrayType],
         s"JacksonGenerator only supports to be initialized with a ${StructType.simpleString} " +
    --- End diff --
    
    maybe need a `,` between `${StructType.simpleString}` and `${MapType.simpleString}`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-24391][SQL] Support arrays of any types by...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r212918588
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonUtils.scala ---
    @@ -32,29 +32,29 @@ object JacksonUtils {
         }
       }
     
    -  /**
    -   * Verify if the schema is supported in JSON parsing.
    -   */
    -  def verifySchema(schema: StructType): Unit = {
    -    def verifyType(name: String, dataType: DataType): Unit = dataType match {
    -      case NullType | BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType |
    -           DoubleType | StringType | TimestampType | DateType | BinaryType | _: DecimalType =>
    +  def verifyType(name: String, dataType: DataType): Unit = dataType match {
    --- End diff --
    
    private?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214844675
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -685,33 +679,29 @@ case class StructsToJson(
             (row: Any) =>
    --- End diff --
    
    `child.dataType` -> `intputSchema`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95235/testReport)** for PR 22226 at commit [`c1755de`](https://github.com/apache/spark/commit/c1755decf93c503dcb3de957408908177a75cf3e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-24391][SQL] Support arrays of any types by...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r212918991
  
    --- Diff: sql/core/src/test/resources/sql-tests/inputs/json-functions.sql ---
    @@ -51,3 +51,8 @@ select from_json('[null, {"a":2}]', 'array<struct<a:int>>');
     
     select from_json('[{"a": 1}, {"b":2}]', 'array<map<string,int>>');
     select from_json('[{"a": 1}, 2]', 'array<map<string,int>>');
    +
    +-- to_json - array type
    +select to_json(array('1','2','3'));
    +select to_json(array(array(1,2,3),array(4)));
    +
    --- End diff --
    
    super nit: add space afiter `,`, e.g., `select to_json(array('1', '2', '3'));`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Seems okay but I or someone else should take a closer look before getting this in.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/22226


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95291/testReport)** for PR 22226 at commit [`906a301`](https://github.com/apache/spark/commit/906a3013e97f8e1d1f8f7e0e335f7404de47b582).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    @HyukjinKwon Please, have a look at the PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95575/testReport)** for PR 22226 at commit [`ce3b17c`](https://github.com/apache/spark/commit/ce3b17c9d0525c6bc27e1ded066cd7e43b597eb1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214156048
  
    --- Diff: R/pkg/R/functions.R ---
    @@ -1697,8 +1697,8 @@ setMethod("to_date",
               })
     
     #' @details
    -#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
    -#' a \code{mapType} or array of \code{mapType} into a Column of JSON string.
    +#' \code{to_json}: Converts a column containing a \code{structType}, a \code{mapType}
    +#' or an array into a Column of JSON string.
    --- End diff --
    
    does `\code{arrayType}` work here? 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95622/testReport)** for PR 22226 at commit [`72d2628`](https://github.com/apache/spark/commit/72d2628323af4e44da1083c99c0d4996c34e4c8c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95496/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95227/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95671/testReport)** for PR 22226 at commit [`90c9687`](https://github.com/apache/spark/commit/90c968772d74c8aadc7a4d0e74e226554b921486).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214243115
  
    --- Diff: R/pkg/R/functions.R ---
    @@ -1697,8 +1697,8 @@ setMethod("to_date",
               })
     
     #' @details
    -#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
    -#' a \code{mapType} or array of \code{mapType} into a Column of JSON string.
    +#' \code{to_json}: Converts a column containing a \code{structType}, a \code{mapType}
    +#' or an array into a Column of JSON string.
    --- End diff --
    
    it should
    could we add some tests for this in R?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95331/testReport)** for PR 22226 at commit [`e91a33c`](https://github.com/apache/spark/commit/e91a33cbf5416c93b95ca0bd594a5418ee033f15).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    @MaxGekk btw, why did you attach this pr to the resolved jira?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214188879
  
    --- Diff: R/pkg/R/functions.R ---
    @@ -1697,8 +1697,8 @@ setMethod("to_date",
               })
     
     #' @details
    -#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
    -#' a \code{mapType} or array of \code{mapType} into a Column of JSON string.
    +#' \code{to_json}: Converts a column containing a \code{structType}, a \code{mapType}
    +#' or an array into a Column of JSON string.
    --- End diff --
    
    I am not sure


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214243817
  
    --- Diff: R/pkg/R/functions.R ---
    @@ -1697,8 +1697,8 @@ setMethod("to_date",
               })
     
     #' @details
    -#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
    -#' a \code{mapType} or array of \code{mapType} into a Column of JSON string.
    +#' \code{to_json}: Converts a column containing a \code{structType}, a \code{mapType}
    +#' or an array into a Column of JSON string.
    --- End diff --
    
    Let's add one simple python doctest as well


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95347/testReport)** for PR 22226 at commit [`2c3a641`](https://github.com/apache/spark/commit/2c3a641cb68a9a414f0e0664d60ba57ef121c325).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95235/testReport)** for PR 22226 at commit [`c1755de`](https://github.com/apache/spark/commit/c1755decf93c503dcb3de957408908177a75cf3e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95671/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214509654
  
    --- Diff: R/pkg/R/functions.R ---
    @@ -1697,8 +1697,8 @@ setMethod("to_date",
               })
     
     #' @details
    -#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
    -#' a \code{mapType} or array of \code{mapType} into a Column of JSON string.
    +#' \code{to_json}: Converts a column containing a \code{structType}, a \code{mapType}
    +#' or an array into a Column of JSON string.
    --- End diff --
    
    I added tests for Python and R. Please, take a look at them.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214572336
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonUtils.scala ---
    @@ -32,29 +32,29 @@ object JacksonUtils {
         }
       }
     
    -  /**
    -   * Verify if the schema is supported in JSON parsing.
    -   */
    -  def verifySchema(schema: StructType): Unit = {
    -    def verifyType(name: String, dataType: DataType): Unit = dataType match {
    -      case NullType | BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType |
    -           DoubleType | StringType | TimestampType | DateType | BinaryType | _: DecimalType =>
    +  def verifyType(name: String, dataType: DataType): Unit = dataType match {
    --- End diff --
    
    We can do:
    
    ```
    def verifyType(name: String, dataType: DataType): Unit = {
      dataType match {
        case ...
      }
    }
    ```
    
    to reduce the diff.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-24391][SQL] Support arrays of any types by...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r212917164
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
    @@ -65,6 +66,8 @@ private[sql] class JacksonGenerator(
           (arr: SpecializedGetters, i: Int) => {
             writeObject(writeMapData(arr.getMap(i), mt, mapElementWriter))
           }
    +    case _ => throw new UnsupportedOperationException(
    +      s"Initial type ${dataType.catalogString} must be an array, a struct or a map")
    --- End diff --
    
    nit: `s"Initial type ${dataType.catalogString} must be an array, a struct or a map")` -> `s"Initial type ${dataType.catalogString} must be an ${ArrayType.simpleString}, a ${StructType.simpleString} or a ${MapType.simpleString}")`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95227/testReport)** for PR 22226 at commit [`e9a5f50`](https://github.com/apache/spark/commit/e9a5f50720d0e6ab301afaa3a4eff801de1240d7).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214157587
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
    @@ -469,4 +469,53 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
     
         checkAnswer(sql("""select json[0] from jsonTable"""), Seq(Row(null)))
       }
    +
    +  test("to_json - array of primitive type") {
    --- End diff --
    
    typo: `primitive type` -> `primitive types`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Probably, you'd be better to file separate jira for each function.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-24391][SQL] Support arrays of any types by...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r212996418
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonUtils.scala ---
    @@ -32,29 +32,29 @@ object JacksonUtils {
         }
       }
     
    -  /**
    -   * Verify if the schema is supported in JSON parsing.
    -   */
    -  def verifySchema(schema: StructType): Unit = {
    -    def verifyType(name: String, dataType: DataType): Unit = dataType match {
    -      case NullType | BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType |
    -           DoubleType | StringType | TimestampType | DateType | BinaryType | _: DecimalType =>
    +  def verifyType(name: String, dataType: DataType): Unit = dataType match {
    --- End diff --
    
    I extracted it to use outside


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214572178
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
    @@ -43,20 +42,22 @@ private[sql] class JacksonGenerator(
       // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
       private type ValueWriter = (SpecializedGetters, Int) => Unit
     
    -  // `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
    -  require(dataType.isInstanceOf[StructType] || dataType.isInstanceOf[MapType],
    -    s"JacksonGenerator only supports to be initialized with a ${StructType.simpleString} " +
    -      s"or ${MapType.simpleString} but got ${dataType.catalogString}")
    +  // `JackGenerator` can only be initialized with a `StructType`, a `MapType` or a `ArrayType`.
    +  require(dataType.isInstanceOf[StructType] || dataType.isInstanceOf[MapType]
    +    || dataType.isInstanceOf[ArrayType],
    +    s"JacksonGenerator only supports to be initialized with a ${StructType.simpleString}, " +
    +      s"${MapType.simpleString} or ${ArrayType.simpleString} but got ${dataType.catalogString}")
     
       // `ValueWriter`s for all fields of the schema
       private lazy val rootFieldWriters: Array[ValueWriter] = dataType match {
         case st: StructType => st.map(_.dataType).map(makeWriter).toArray
         case _ => throw new UnsupportedOperationException(
    -      s"Initial type ${dataType.catalogString} must be a struct")
    +      s"Initial type ${dataType.catalogString} must be a ${StructType.simpleString}")
       }
     
       // `ValueWriter` for array data storing rows of the schema.
       private lazy val arrElementWriter: ValueWriter = dataType match {
    +    case at: ArrayType => makeWriter(at.elementType)
         case st: StructType =>
    --- End diff --
    
    Can we do `case _: StructType | _: MapType => makeWriter(dataType)`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95291/testReport)** for PR 22226 at commit [`906a301`](https://github.com/apache/spark/commit/906a3013e97f8e1d1f8f7e0e335f7404de47b582).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95347 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95347/testReport)** for PR 22226 at commit [`2c3a641`](https://github.com/apache/spark/commit/2c3a641cb68a9a414f0e0664d60ba57ef121c325).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95331/testReport)** for PR 22226 at commit [`e91a33c`](https://github.com/apache/spark/commit/e91a33cbf5416c93b95ca0bd594a5418ee033f15).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95496/testReport)** for PR 22226 at commit [`2f73c9c`](https://github.com/apache/spark/commit/2f73c9c4d3cef21997522563d1f7dcca58f8e9cd).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214843579
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -685,33 +679,29 @@ case class StructsToJson(
             (row: Any) =>
               gen.write(row.asInstanceOf[InternalRow])
               getAndReset()
    -      case ArrayType(_: StructType, _) =>
    +      case _: ArrayType =>
             (arr: Any) =>
               gen.write(arr.asInstanceOf[ArrayData])
               getAndReset()
           case _: MapType =>
             (map: Any) =>
               gen.write(map.asInstanceOf[MapData])
               getAndReset()
    -      case ArrayType(_: MapType, _) =>
    -        (arr: Any) =>
    -          gen.write(arr.asInstanceOf[ArrayData])
    -          getAndReset()
         }
       }
     
       override def dataType: DataType = StringType
     
       override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
    -    case _: StructType | ArrayType(_: StructType, _) =>
    +    case _: StructType =>
    --- End diff --
    
    nit: `case _: StructType` and use it instead of `rowSchema.asInstanceOf[StructType]`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95496/testReport)** for PR 22226 at commit [`2f73c9c`](https://github.com/apache/spark/commit/2f73c9c4d3cef21997522563d1f7dcca58f8e9cd).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    **[Test build #95575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95575/testReport)** for PR 22226 at commit [`ce3b17c`](https://github.com/apache/spark/commit/ce3b17c9d0525c6bc27e1ded066cd7e43b597eb1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214843691
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -685,33 +679,29 @@ case class StructsToJson(
             (row: Any) =>
               gen.write(row.asInstanceOf[InternalRow])
               getAndReset()
    -      case ArrayType(_: StructType, _) =>
    +      case _: ArrayType =>
             (arr: Any) =>
               gen.write(arr.asInstanceOf[ArrayData])
               getAndReset()
           case _: MapType =>
             (map: Any) =>
               gen.write(map.asInstanceOf[MapData])
               getAndReset()
    -      case ArrayType(_: MapType, _) =>
    -        (arr: Any) =>
    -          gen.write(arr.asInstanceOf[ArrayData])
    -          getAndReset()
         }
       }
     
       override def dataType: DataType = StringType
     
       override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
    -    case _: StructType | ArrayType(_: StructType, _) =>
    +    case _: StructType =>
           try {
             JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
             TypeCheckResult.TypeCheckSuccess
           } catch {
             case e: UnsupportedOperationException =>
               TypeCheckResult.TypeCheckFailure(e.getMessage)
           }
    -    case _: MapType | ArrayType(_: MapType, _) =>
    +    case _: MapType =>
    --- End diff --
    
    nit `case mapType: Maptype =>` and use it below likewise.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-24391][SQL] Support arrays of any types by to_jso...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    cc @dongjoon-hyun Try to review this PR?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214155095
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
    @@ -28,7 +27,7 @@ import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
     import org.apache.spark.sql.types._
     
     /**
    - * `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
    + * `JackGenerator` can only be initialized with a `StructType`, a `MapType` ot `ArrayType`.
    --- End diff --
    
    typo: `ot`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95575/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95347/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214842392
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -663,12 +662,7 @@ case class StructsToJson(
         rowSchema, writer, new JSONOptions(options, timeZoneId.get))
     
       @transient
    -  lazy val rowSchema = child.dataType match {
    -    case st: StructType => st
    -    case ArrayType(st: StructType, _) => st
    -    case mt: MapType => mt
    -    case ArrayType(mt: MapType, _) => mt
    -  }
    +  lazy val rowSchema = child.dataType
    --- End diff --
    
    nit: `rowSchema` -> `intputSchema`. I named this to `rowSchema` because it was always the schema for the row itself. Now, it seems can be other types as well.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-24391][SQL] Support arrays of any types by...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r212995984
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
    @@ -65,6 +66,8 @@ private[sql] class JacksonGenerator(
           (arr: SpecializedGetters, i: Int) => {
             writeObject(writeMapData(arr.getMap(i), mt, mapElementWriter))
           }
    +    case _ => throw new UnsupportedOperationException(
    +      s"Initial type ${dataType.catalogString} must be an array, a struct or a map")
    --- End diff --
    
    Changed here and above


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r214848902
  
    --- Diff: R/pkg/R/functions.R ---
    @@ -1697,8 +1697,8 @@ setMethod("to_date",
               })
     
     #' @details
    -#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
    -#' a \code{mapType} or array of \code{mapType} into a Column of JSON string.
    +#' \code{to_json}: Converts a column containing a \code{structType}, a \code{mapType}
    +#' or an array into a Column of JSON string.
    --- End diff --
    
    `\code{arrayType}`. It seems missed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    > Probably, you'd be better to file separate jira for each function.
    > +1 for separate JIRA.
    
    I created the JIRA ticket: https://issues.apache.org/jira/browse/SPARK-25252



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95622/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22226
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95331/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22226#discussion_r213177782
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
    @@ -613,8 +613,7 @@ case class JsonToStructs(
     }
     
     /**
    - * Converts a [[StructType]], [[ArrayType]] of [[StructType]]s, [[MapType]]
    - * or [[ArrayType]] of [[MapType]]s to a json output string.
    + * Converts a [[StructType]], [[ArrayType]] or [[MapType]] to a json output string.
    --- End diff --
    
    not a big deal but `JSON` while we are here


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org