You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gengliangwang <gi...@git.apache.org> on 2018/07/22 12:43:06 UTC
[GitHub] spark pull request #21838: [SPARK-24811][SQL]Avro: add new function from_avr...
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/21838
[SPARK-24811][SQL]Avro: add new function from_avro and to_avro
## What changes were proposed in this pull request?
1. Add a new function from_avro for parsing a binary column of avro format and converting it into its corresponding catalyst value.
2. Add a new function to_avro for converting a column into binary of avro format with the specified schema.
I created #21774 for this, but it failed the build https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-maven-hadoop-2.6/7902/
Additional changes In this PR:
1. Add `scalacheck` dependency in pom.xml to resolve the failure.
2. Update the `log4j.properties` to make it consistent with other modules.
## How was this patch tested?
Unit test
Compile with different commands:
```
./build/mvn --force -DzincPort=3643 -DskipTests -Phadoop-2.6 -Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pmesos -Pyarn compile test-compile
./build/mvn --force -DzincPort=3643 -DskipTests -Phadoop-2.7 -Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pmesos -Pyarn compile test-compile
./build/mvn --force -DzincPort=3643 -DskipTests -Phadoop-3.1 -Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pmesos -Pyarn compile test-compile
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark from_and_to_avro
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21838.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21838
----
commit b02d7edbf39f9ad4c4a2fdae6eeec1104097057e
Author: Gengliang Wang <ge...@...>
Date: 2018-07-15T18:47:29Z
add from_avro and to_avro
commit 8aa2b31fcfa7d3a36e47f5c0939c575971e95df3
Author: Gengliang Wang <ge...@...>
Date: 2018-07-17T14:35:27Z
add test suite
commit 54861323bec556c66e94ca9919760ac9b3a3c11e
Author: Gengliang Wang <ge...@...>
Date: 2018-07-18T09:43:27Z
add AvroFunctionsSuite
commit 76763da105ae6b6d05802c7ce846981ee9b29d3b
Author: Gengliang Wang <ge...@...>
Date: 2018-07-19T07:28:01Z
address some comments
commit 81614c23de0b1b7717b5b61dc81ab15e51671f3e
Author: Gengliang Wang <ge...@...>
Date: 2018-07-19T16:37:44Z
codegen
commit e5c170281429e0a3b2981eef9c08a2a757c89d0d
Author: Gengliang Wang <ge...@...>
Date: 2018-07-19T19:01:54Z
address comments
commit 8421f445160eac368a90f21ef091bdf1101eb390
Author: Gengliang Wang <ge...@...>
Date: 2018-07-20T07:45:31Z
address comments
commit 47caecee4efdfae4b74c05974f36254d2974c278
Author: Gengliang Wang <ge...@...>
Date: 2018-07-20T08:24:45Z
override sql
commit f54adc9251936abecf1cd8403c7df6b0d9b5a71e
Author: Gengliang Wang <ge...@...>
Date: 2018-07-22T12:31:13Z
add scalacheck and fix build failure
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21838
**[Test build #93410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93410/testReport)** for PR 21838 at commit [`f54adc9`](https://github.com/apache/spark/commit/f54adc9251936abecf1cd8403c7df6b0d9b5a71e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21838
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21838
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93410/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21838: [SPARK-24811][SQL]Avro: add new function from_avr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21838
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21838
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21838
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1215/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21838: [SPARK-24811][SQL]Avro: add new function from_avr...
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/21838#discussion_r234099158
--- Diff: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroCatalystDataConversionSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.{AvroDataToCatalyst, CatalystDataToAvro, RandomDataGenerator}
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.{ExpressionEvalHelper, GenericInternalRow, Literal}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData, MapData}
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+class AvroCatalystDataConversionSuite extends SparkFunSuite with ExpressionEvalHelper {
+
+ private def roundTripTest(data: Literal): Unit = {
+ val avroType = SchemaConverters.toAvroType(data.dataType, data.nullable)
+ checkResult(data, avroType.toString, data.eval())
+ }
+
+ private def checkResult(data: Literal, schema: String, expected: Any): Unit = {
+ checkEvaluation(
+ AvroDataToCatalyst(CatalystDataToAvro(data), schema),
+ prepareExpectedResult(expected))
+ }
+
+ private def assertFail(data: Literal, schema: String): Unit = {
+ intercept[java.io.EOFException] {
+ AvroDataToCatalyst(CatalystDataToAvro(data), schema).eval()
+ }
+ }
+
+ private val testingTypes = Seq(
+ BooleanType,
+ ByteType,
+ ShortType,
+ IntegerType,
+ LongType,
+ FloatType,
+ DoubleType,
+ DecimalType(8, 0), // 32 bits decimal without fraction
+ DecimalType(8, 4), // 32 bits decimal
+ DecimalType(16, 0), // 64 bits decimal without fraction
+ DecimalType(16, 11), // 64 bits decimal
+ DecimalType(38, 0),
+ DecimalType(38, 38),
+ StringType,
+ BinaryType)
+
+ protected def prepareExpectedResult(expected: Any): Any = expected match {
+ // Spark decimal is converted to avro string=
+ case d: Decimal => UTF8String.fromString(d.toString)
+ // Spark byte and short both map to avro int
+ case b: Byte => b.toInt
+ case s: Short => s.toInt
+ case row: GenericInternalRow => InternalRow.fromSeq(row.values.map(prepareExpectedResult))
+ case array: GenericArrayData => new GenericArrayData(array.array.map(prepareExpectedResult))
+ case map: MapData =>
+ val keys = new GenericArrayData(
+ map.keyArray().asInstanceOf[GenericArrayData].array.map(prepareExpectedResult))
+ val values = new GenericArrayData(
+ map.valueArray().asInstanceOf[GenericArrayData].array.map(prepareExpectedResult))
+ new ArrayBasedMapData(keys, values)
+ case other => other
+ }
+
+ testingTypes.foreach { dt =>
+ val seed = scala.util.Random.nextLong()
+ test(s"single $dt with seed $seed") {
+ val rand = new scala.util.Random(seed)
+ val data = RandomDataGenerator.forType(dt, rand = rand).get.apply()
+ val converter = CatalystTypeConverters.createToCatalystConverter(dt)
+ val input = Literal.create(converter(data), dt)
+ roundTripTest(input)
+ }
+ }
+
+ for (_ <- 1 to 5) {
--- End diff --
Why not `(1 to 5).foreach`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21838
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21838: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21838
**[Test build #93410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93410/testReport)** for PR 21838 at commit [`f54adc9`](https://github.com/apache/spark/commit/f54adc9251936abecf1cd8403c7df6b0d9b5a71e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org