You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2016/07/02 12:27:49 UTC

[GitHub] spark pull request #14028: [SPARK-16351][SQL] Avoid record-per type dispatch...

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/14028

    [SPARK-16351][SQL] Avoid record-per type dispatch in JSON when writing

    ## What changes were proposed in this pull request?
    
    Currently, `JacksonGenerator.apply` is doing type-based dispatch for each row to write appropriate values.
    It might not have to be done like this because the schema is already kept.
    
    So, appropriate writers can be created first according to the schema, and then apply them to each row. This approach is similar with `CatalystWriteSupport`.
    
    This PR corrects `JacksonGenerator` so that it creates some writes for the schema and then applies them to each row rather than every time.
    
    Benchmark was proceeded with the codes below:
    
    ```
    test("Benchmark for JSON writer") {
      val N = 500 << 8
      val row =
        """{"struct":{"field1": true, "field2": 92233720368547758070},
            "structWithArrayFields":{"field1":[4, 5, 6], "field2":["str1", "str2"]},
            "arrayOfString":["str1", "str2"],
            "arrayOfInteger":[1, 2147483647, -2147483648],
            "arrayOfLong":[21474836470, 9223372036854775807, -9223372036854775808],
            "arrayOfBigInteger":[922337203685477580700, -922337203685477580800],
            "arrayOfDouble":[1.2, 1.7976931348623157E308, 4.9E-324, 2.2250738585072014E-308],
            "arrayOfBoolean":[true, false, true],
            "arrayOfNull":[null, null, null, null],
            "arrayOfStruct":[{"field1": true, "field2": "str1"}, {"field1": false}, {"field3": null}],
            "arrayOfArray1":[[1, 2, 3], ["str1", "str2"]],
            "arrayOfArray2":[[1, 2, 3], [1.1, 2.1, 3.1]]
           }"""
      val df = spark.sqlContext.read.json(spark.sparkContext.parallelize(List.fill(N)(row)))
      (0 to 10).foreach { _ =>
        val benchmark = new Benchmark("JSON writer", N)
        benchmark.addCase("writing JSON file", 10) { iter =>
          withTempPath { path =>
            df.write.format("json").save(path.getCanonicalPath)
          }
        }
        benchmark.run()
      }
    }
    ```
    
    This produced the results below
    
    - **Before**
    
    ```
    JSON writer:                             Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    writing JSON file                             1675 / 1767          0.1       13087.5       1.0X
    ```
    
    - **After**
    
    ```
    JSON writer:                             Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    writing JSON file                             1597 / 1686          0.1       12477.1       1.0X
    ```
    
    In addition, I ran this benchmarks 10 times for each and calculated the average times as below:
    
    | **Before** | **After**|
    |---------------|------------|
    |17478ms  |16669ms |
    
    
    It seems roughly ~5% is improved.
    
    ## How was this patch tested?
    
    Existing tests should cover this.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark SPARK-16351

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/14028.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14028
    
----
commit 76ca05a383b1b3f6deea266276af872ab3a18f36
Author: hyukjinkwon <gu...@gmail.com>
Date:   2016-07-02T12:16:25Z

    Avoid record-per type dispatch in JSON when writing

commit 34ec476b5afad926db89a03ff64e6cda8263ee86
Author: hyukjinkwon <gu...@gmail.com>
Date:   2016-07-02T12:18:03Z

    Keep the comment

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14028: [SPARK-16351][SQL] Avoid per-record type dispatch...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14028#discussion_r69936163
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonGenerator.scala ---
    @@ -17,74 +17,180 @@
     
     package org.apache.spark.sql.execution.datasources.json
     
    +import java.io.Writer
    +
     import com.fasterxml.jackson.core._
     
     import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
     import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
     import org.apache.spark.sql.types._
     
    -private[sql] object JacksonGenerator {
    -  /** Transforms a single InternalRow to JSON using Jackson
    -   *
    -   * TODO: make the code shared with the other apply method.
    -   *
    -   * @param rowSchema the schema object used for conversion
    -   * @param gen a JsonGenerator object
    -   * @param row The row to convert
    -   */
    -  def apply(rowSchema: StructType, gen: JsonGenerator)(row: InternalRow): Unit = {
    -    def valWriter: (DataType, Any) => Unit = {
    -      case (_, null) | (NullType, _) => gen.writeNull()
    -      case (StringType, v) => gen.writeString(v.toString)
    -      case (TimestampType, v: Long) => gen.writeString(DateTimeUtils.toJavaTimestamp(v).toString)
    -      case (IntegerType, v: Int) => gen.writeNumber(v)
    -      case (ShortType, v: Short) => gen.writeNumber(v)
    -      case (FloatType, v: Float) => gen.writeNumber(v)
    -      case (DoubleType, v: Double) => gen.writeNumber(v)
    -      case (LongType, v: Long) => gen.writeNumber(v)
    -      case (DecimalType(), v: Decimal) => gen.writeNumber(v.toJavaBigDecimal)
    -      case (ByteType, v: Byte) => gen.writeNumber(v.toInt)
    -      case (BinaryType, v: Array[Byte]) => gen.writeBinary(v)
    -      case (BooleanType, v: Boolean) => gen.writeBoolean(v)
    -      case (DateType, v: Int) => gen.writeString(DateTimeUtils.toJavaDate(v).toString)
    -      // For UDT values, they should be in the SQL type's corresponding value type.
    -      // We should not see values in the user-defined class at here.
    -      // For example, VectorUDT's SQL type is an array of double. So, we should expect that v is
    -      // an ArrayData at here, instead of a Vector.
    -      case (udt: UserDefinedType[_], v) => valWriter(udt.sqlType, v)
    -
    -      case (ArrayType(ty, _), v: ArrayData) =>
    -        gen.writeStartArray()
    -        v.foreach(ty, (_, value) => valWriter(ty, value))
    -        gen.writeEndArray()
    -
    -      case (MapType(kt, vt, _), v: MapData) =>
    -        gen.writeStartObject()
    -        v.foreach(kt, vt, { (k, v) =>
    -          gen.writeFieldName(k.toString)
    -          valWriter(vt, v)
    -        })
    -        gen.writeEndObject()
    -
    -      case (StructType(ty), v: InternalRow) =>
    -        gen.writeStartObject()
    -        var i = 0
    -        while (i < ty.length) {
    -          val field = ty(i)
    -          val value = v.get(i, field.dataType)
    -          if (value != null) {
    -            gen.writeFieldName(field.name)
    -            valWriter(field.dataType, value)
    -          }
    -          i += 1
    +private[sql] class JacksonGenerator(schema: StructType, writer: Writer) {
    +  // A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
    +  // JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
    +  // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
    +  private type ValueWriter = (SpecializedGetters, Int) => Unit
    +
    +  // `ValueWriter`s for all fields of the schema
    +  private val rootFieldWriters: Seq[ValueWriter] = schema.map(_.dataType).map(makeWriter)
    +
    +  private val gen = new JsonFactory().createGenerator(writer).setRootValueSeparator(null)
    +
    +  private def makeWriter(dataType: DataType): ValueWriter = dataType match {
    +    case NullType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNull()
    +
    +    case BooleanType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeBoolean(row.getBoolean(ordinal))
    +
    +    case ByteType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getByte(ordinal))
    +
    +    case ShortType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getShort(ordinal))
    +
    +    case IntegerType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getInt(ordinal))
    +
    +    case LongType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getLong(ordinal))
    +
    +    case FloatType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getFloat(ordinal))
    +
    +    case DoubleType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getDouble(ordinal))
    +
    +    case StringType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeString(row.getUTF8String(ordinal).toString)
    +
    +    case TimestampType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeString(DateTimeUtils.toJavaTimestamp(row.getLong(ordinal)).toString)
    +
    +    case DateType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeString(DateTimeUtils.toJavaDate(row.getInt(ordinal)).toString)
    +
    +    case BinaryType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeBinary(row.getBinary(ordinal))
    +
    +    case dt: DecimalType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getDecimal(ordinal, dt.precision, dt.scale).toJavaBigDecimal)
    +
    +    case st: StructType =>
    +      val fieldWriters = st.map(_.dataType).map(makeWriter)
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        writeObject {
    +          writeFields(row.getStruct(ordinal, st.length), st, fieldWriters)
    +        }
    +
    +    case at: ArrayType =>
    +      val elementWriter = makeWriter(at.elementType)
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        writeArray {
    +          writeArrayData(row.getArray(ordinal), elementWriter)
    --- End diff --
    
    Inline both writeArray and writeArrayData?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Thanks @yhuai! I just addressed your comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61654/consoleFull)** for PR 14028 at commit [`93aea42`](https://github.com/apache/spark/commit/93aea42fc86e69eedf4f35f5682b35744e12a480).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62445/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61654/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61656/consoleFull)** for PR 14028 at commit [`50b6c4c`](https://github.com/apache/spark/commit/50b6c4cdfae12151262a83b9283acbfaf2b05938).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61943/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61656/consoleFull)** for PR 14028 at commit [`50b6c4c`](https://github.com/apache/spark/commit/50b6c4cdfae12151262a83b9283acbfaf2b05938).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    @yhuai Could you take another look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61657/consoleFull)** for PR 14028 at commit [`e8edd38`](https://github.com/apache/spark/commit/e8edd38a1b98f5f886b9e8f29ab181fefe2e7202).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #62445 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62445/consoleFull)** for PR 14028 at commit [`6570a98`](https://github.com/apache/spark/commit/6570a9874e60ecb9366ea37a0e5dfe06b821dc62).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    BTW, this approach can be applied to reading in JSON and CSV as well if this change is appropriate. I will work on them if this change is sensible and can be merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14028: [SPARK-16351][SQL] Avoid per-record type dispatch...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14028#discussion_r69936226
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonGenerator.scala ---
    @@ -17,74 +17,180 @@
     
     package org.apache.spark.sql.execution.datasources.json
     
    +import java.io.Writer
    +
     import com.fasterxml.jackson.core._
     
     import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
     import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
     import org.apache.spark.sql.types._
     
    -private[sql] object JacksonGenerator {
    -  /** Transforms a single InternalRow to JSON using Jackson
    -   *
    -   * TODO: make the code shared with the other apply method.
    -   *
    -   * @param rowSchema the schema object used for conversion
    -   * @param gen a JsonGenerator object
    -   * @param row The row to convert
    -   */
    -  def apply(rowSchema: StructType, gen: JsonGenerator)(row: InternalRow): Unit = {
    -    def valWriter: (DataType, Any) => Unit = {
    -      case (_, null) | (NullType, _) => gen.writeNull()
    -      case (StringType, v) => gen.writeString(v.toString)
    -      case (TimestampType, v: Long) => gen.writeString(DateTimeUtils.toJavaTimestamp(v).toString)
    -      case (IntegerType, v: Int) => gen.writeNumber(v)
    -      case (ShortType, v: Short) => gen.writeNumber(v)
    -      case (FloatType, v: Float) => gen.writeNumber(v)
    -      case (DoubleType, v: Double) => gen.writeNumber(v)
    -      case (LongType, v: Long) => gen.writeNumber(v)
    -      case (DecimalType(), v: Decimal) => gen.writeNumber(v.toJavaBigDecimal)
    -      case (ByteType, v: Byte) => gen.writeNumber(v.toInt)
    -      case (BinaryType, v: Array[Byte]) => gen.writeBinary(v)
    -      case (BooleanType, v: Boolean) => gen.writeBoolean(v)
    -      case (DateType, v: Int) => gen.writeString(DateTimeUtils.toJavaDate(v).toString)
    -      // For UDT values, they should be in the SQL type's corresponding value type.
    -      // We should not see values in the user-defined class at here.
    -      // For example, VectorUDT's SQL type is an array of double. So, we should expect that v is
    -      // an ArrayData at here, instead of a Vector.
    -      case (udt: UserDefinedType[_], v) => valWriter(udt.sqlType, v)
    -
    -      case (ArrayType(ty, _), v: ArrayData) =>
    -        gen.writeStartArray()
    -        v.foreach(ty, (_, value) => valWriter(ty, value))
    -        gen.writeEndArray()
    -
    -      case (MapType(kt, vt, _), v: MapData) =>
    -        gen.writeStartObject()
    -        v.foreach(kt, vt, { (k, v) =>
    -          gen.writeFieldName(k.toString)
    -          valWriter(vt, v)
    -        })
    -        gen.writeEndObject()
    -
    -      case (StructType(ty), v: InternalRow) =>
    -        gen.writeStartObject()
    -        var i = 0
    -        while (i < ty.length) {
    -          val field = ty(i)
    -          val value = v.get(i, field.dataType)
    -          if (value != null) {
    -            gen.writeFieldName(field.name)
    -            valWriter(field.dataType, value)
    -          }
    -          i += 1
    +private[sql] class JacksonGenerator(schema: StructType, writer: Writer) {
    +  // A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
    +  // JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
    +  // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
    +  private type ValueWriter = (SpecializedGetters, Int) => Unit
    +
    +  // `ValueWriter`s for all fields of the schema
    +  private val rootFieldWriters: Seq[ValueWriter] = schema.map(_.dataType).map(makeWriter)
    --- End diff --
    
    Let's use an array. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61943/consoleFull)** for PR 14028 at commit [`6570a98`](https://github.com/apache/spark/commit/6570a9874e60ecb9366ea37a0e5dfe06b821dc62).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14028: [SPARK-16351][SQL] Avoid per-record type dispatch...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14028#discussion_r69936170
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonGenerator.scala ---
    @@ -17,74 +17,180 @@
     
     package org.apache.spark.sql.execution.datasources.json
     
    +import java.io.Writer
    +
     import com.fasterxml.jackson.core._
     
     import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
     import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
     import org.apache.spark.sql.types._
     
    -private[sql] object JacksonGenerator {
    -  /** Transforms a single InternalRow to JSON using Jackson
    -   *
    -   * TODO: make the code shared with the other apply method.
    -   *
    -   * @param rowSchema the schema object used for conversion
    -   * @param gen a JsonGenerator object
    -   * @param row The row to convert
    -   */
    -  def apply(rowSchema: StructType, gen: JsonGenerator)(row: InternalRow): Unit = {
    -    def valWriter: (DataType, Any) => Unit = {
    -      case (_, null) | (NullType, _) => gen.writeNull()
    -      case (StringType, v) => gen.writeString(v.toString)
    -      case (TimestampType, v: Long) => gen.writeString(DateTimeUtils.toJavaTimestamp(v).toString)
    -      case (IntegerType, v: Int) => gen.writeNumber(v)
    -      case (ShortType, v: Short) => gen.writeNumber(v)
    -      case (FloatType, v: Float) => gen.writeNumber(v)
    -      case (DoubleType, v: Double) => gen.writeNumber(v)
    -      case (LongType, v: Long) => gen.writeNumber(v)
    -      case (DecimalType(), v: Decimal) => gen.writeNumber(v.toJavaBigDecimal)
    -      case (ByteType, v: Byte) => gen.writeNumber(v.toInt)
    -      case (BinaryType, v: Array[Byte]) => gen.writeBinary(v)
    -      case (BooleanType, v: Boolean) => gen.writeBoolean(v)
    -      case (DateType, v: Int) => gen.writeString(DateTimeUtils.toJavaDate(v).toString)
    -      // For UDT values, they should be in the SQL type's corresponding value type.
    -      // We should not see values in the user-defined class at here.
    -      // For example, VectorUDT's SQL type is an array of double. So, we should expect that v is
    -      // an ArrayData at here, instead of a Vector.
    -      case (udt: UserDefinedType[_], v) => valWriter(udt.sqlType, v)
    -
    -      case (ArrayType(ty, _), v: ArrayData) =>
    -        gen.writeStartArray()
    -        v.foreach(ty, (_, value) => valWriter(ty, value))
    -        gen.writeEndArray()
    -
    -      case (MapType(kt, vt, _), v: MapData) =>
    -        gen.writeStartObject()
    -        v.foreach(kt, vt, { (k, v) =>
    -          gen.writeFieldName(k.toString)
    -          valWriter(vt, v)
    -        })
    -        gen.writeEndObject()
    -
    -      case (StructType(ty), v: InternalRow) =>
    -        gen.writeStartObject()
    -        var i = 0
    -        while (i < ty.length) {
    -          val field = ty(i)
    -          val value = v.get(i, field.dataType)
    -          if (value != null) {
    -            gen.writeFieldName(field.name)
    -            valWriter(field.dataType, value)
    -          }
    -          i += 1
    +private[sql] class JacksonGenerator(schema: StructType, writer: Writer) {
    +  // A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
    +  // JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
    +  // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
    +  private type ValueWriter = (SpecializedGetters, Int) => Unit
    +
    +  // `ValueWriter`s for all fields of the schema
    +  private val rootFieldWriters: Seq[ValueWriter] = schema.map(_.dataType).map(makeWriter)
    +
    +  private val gen = new JsonFactory().createGenerator(writer).setRootValueSeparator(null)
    +
    +  private def makeWriter(dataType: DataType): ValueWriter = dataType match {
    +    case NullType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNull()
    +
    +    case BooleanType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeBoolean(row.getBoolean(ordinal))
    +
    +    case ByteType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getByte(ordinal))
    +
    +    case ShortType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getShort(ordinal))
    +
    +    case IntegerType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getInt(ordinal))
    +
    +    case LongType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getLong(ordinal))
    +
    +    case FloatType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getFloat(ordinal))
    +
    +    case DoubleType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getDouble(ordinal))
    +
    +    case StringType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeString(row.getUTF8String(ordinal).toString)
    +
    +    case TimestampType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeString(DateTimeUtils.toJavaTimestamp(row.getLong(ordinal)).toString)
    +
    +    case DateType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeString(DateTimeUtils.toJavaDate(row.getInt(ordinal)).toString)
    +
    +    case BinaryType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeBinary(row.getBinary(ordinal))
    +
    +    case dt: DecimalType =>
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        gen.writeNumber(row.getDecimal(ordinal, dt.precision, dt.scale).toJavaBigDecimal)
    +
    +    case st: StructType =>
    +      val fieldWriters = st.map(_.dataType).map(makeWriter)
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        writeObject {
    +          writeFields(row.getStruct(ordinal, st.length), st, fieldWriters)
    +        }
    +
    +    case at: ArrayType =>
    +      val elementWriter = makeWriter(at.elementType)
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        writeArray {
    +          writeArrayData(row.getArray(ordinal), elementWriter)
             }
    -        gen.writeEndObject()
     
    -      case (dt, v) =>
    -        sys.error(
    -          s"Failed to convert value $v (class of ${v.getClass}}) with the type of $dt to JSON.")
    +    case mt: MapType =>
    +      val valueWriter = makeWriter(mt.valueType)
    +      (row: SpecializedGetters, ordinal: Int) =>
    +        writeObject {
    +          writeMapData(row.getMap(ordinal), mt, valueWriter)
    --- End diff --
    
    Inline this function?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61657/consoleFull)** for PR 14028 at commit [`e8edd38`](https://github.com/apache/spark/commit/e8edd38a1b98f5f886b9e8f29ab181fefe2e7202).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Sure! Thank toy for your review!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61657/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61653/consoleFull)** for PR 14028 at commit [`34ec476`](https://github.com/apache/spark/commit/34ec476b5afad926db89a03ff64e6cda8263ee86).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    @rxin and @yhuai Could you review this maybe?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61653/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61654/consoleFull)** for PR 14028 at commit [`93aea42`](https://github.com/apache/spark/commit/93aea42fc86e69eedf4f35f5682b35744e12a480).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Actually, I resembled your codes @liancheng here. Would this change be sensible maybe?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61943/consoleFull)** for PR 14028 at commit [`6570a98`](https://github.com/apache/spark/commit/6570a9874e60ecb9366ea37a0e5dfe06b821dc62).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    LGTM, but I'd like to let @yhuai to sign off. Thanks for working on this!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #61653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61653/consoleFull)** for PR 14028 at commit [`34ec476`](https://github.com/apache/spark/commit/34ec476b5afad926db89a03ff64e6cda8263ee86).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    LGTM pending jenkins. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    **[Test build #62445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62445/consoleFull)** for PR 14028 at commit [`6570a98`](https://github.com/apache/spark/commit/6570a9874e60ecb9366ea37a0e5dfe06b821dc62).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Nit: "record-per" in PR title should be "per-record".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14028: [SPARK-16351][SQL] Avoid per-record type dispatch...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/14028


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid record-per type dispatch in JSO...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61656/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org