You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yhuai <gi...@git.apache.org> on 2014/06/06 22:42:25 UTC
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/999
[Spark 2060][SQL] Querying JSON Datasets with SQL and DSL in Spark SQL
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark newJson
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/999.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #999
----
commit f31065f2aac5b5638c3f3a84ee3eb901da816f64
Author: Yin Huai <hu...@cse.ohio-state.edu>
Date: 2014-06-06T19:21:46Z
A query plan or a SchemaRDD can print out its schema.
commit f45583b0f244b6ef9293202333f82bcdc1abcd00
Author: Yin Huai <hu...@cse.ohio-state.edu>
Date: 2014-06-06T20:38:41Z
Infer the schema of a JSON dataset (a text file with one JSON object per line or a RDD[String] with one JSON object per string) and returns a SchemaRDD.
commit af91b23cf843defe8b962611d87ec73a28091300
Author: Yin Huai <hu...@cse.ohio-state.edu>
Date: 2014-06-06T20:40:05Z
Merge remote-tracking branch 'upstream/master' into newJson
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45382166
Pull request #999!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46255972
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15831/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13831773
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala ---
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import org.apache.spark.annotation.{DeveloperApi, Experimental}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.Logging
+
+@Experimental
+private[sql] object JsonRDD extends Logging {
+
+ @DeveloperApi
+ private[sql] def inferSchema(
+ json: RDD[String],
+ samplingRatio: Double = 1.0): LogicalPlan = {
+ require(samplingRatio > 0)
+ val schemaData = if (samplingRatio > 0.99) json else json.sample(false, samplingRatio, 1)
+
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ val baseSchema = createSchema(allKeys)
+
+ createLogicalPlan(json, baseSchema)
+ }
+
+ private def createLogicalPlan(
+ json: RDD[String],
+ baseSchema: StructType): LogicalPlan = {
+ val schema = nullTypeToStringType(baseSchema)
+
+ SparkLogicalPlan(ExistingRdd(asAttributes(schema), parseJson(json).map(asRow(_, schema))))
+ }
+
+ private def createSchema(allKeys: Set[(String, DataType)]): StructType = {
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ (fieldName, dataType)
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq
+
+ StructType(
+ (topLevelFields ++ structFields).sortBy {
+ case StructField(name, _, _) => name
+ })
+ }
+
+ makeStruct(resolved.keySet.toSeq, Nil)
+ }
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ private[json] def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = HiveTypeCoercion.allPromotions.find(p => p.contains(t1) && p
+ .contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ case (StructType(fields1), StructType(fields2)) => {
+ val newFields = (fields1 ++ fields2).groupBy(field => field.name).map {
+ case (name, fieldTypes) => {
+ val dataType = fieldTypes.map(field => field.dataType).reduce(
+ (type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+ StructField(name, dataType, true)
+ }
+ }
+ StructType(newFields.toSeq.sortBy {
+ case StructField(name, _, _) => name
+ })
+ }
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ // TODO: We should use JsonObjectStringType to mark that values of field will be
+ // strings and every string is a Json object.
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ private def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // Unexpected data type.
+ case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ private def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When the value of a key is an JSON object, we
+ * only use a placeholder (StructType(Nil)) to mark that it should be a struct
+ * instead of getting all fields of this struct because a field does not appear
+ * in this JSON object can appear in other JSON objects.
+ */
+ private def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
--- End diff --
There are a few places where we need to do `.asInstanceOf[Map[String, Any]]` or `.asInstanceOf[Seq[Any]]`. So, I think it may be better to convert types of those collections in a row at the same place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46387957
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15862/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45814366
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45637977
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46387956
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46380653
This looks to me overall. Only few nitpicks.
I think we should merge it after you addressed the couple comments I had.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46383326
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45382139
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520493
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
+ val minId = attributes.map(_.exprId.id).min
+ attributes.map {
+ case a: AttributeReference =>
+ AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
+ }
+ }
+
+ protected def checkSchema(expected: Seq[Attribute], actual: Seq[Attribute]): Unit = {
+ val normalizedExpected = normalizeExprIds(expected).toSeq
+ val normalizedActual = normalizeExprIds(actual).toSeq
+ if (normalizedExpected != normalizedActual) {
+ fail(
+ s"""
+ |=== FAIL: Schemas do not match ===
+ |${sideBySide(
+ s"== Expected Schema ==\n" +
+ generateSchemaTreeString(normalizedExpected),
+ s"== Actual Schema ==\n" +
+ generateSchemaTreeString(normalizedActual)).mkString("\n")}
+ """.stripMargin)
+ }
+ }
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
+ // TODO: Re-enable the following test.
+ // Getting all values of a specific field from an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable"),
+ (Seq(true, false), Seq("str1", null)) :: Nil
+ )
+ */
+
+ // Access a struct and fields inside of it.
+ checkAnswer(
+ sql("select struct, struct.field1, struct.field2 from jsonTable"),
+ (
+ Seq(true, BigDecimal("92233720368547758070")),
+ true,
+ BigDecimal("92233720368547758070")) :: Nil
+ )
+
+ // Access an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1, structWithArrayFields.field2 from jsonTable"),
+ (Seq(4, 5, 6), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1[1], structWithArrayFields.field2[3] from jsonTable"),
+ (5, null) :: Nil
+ )
+ }
+
+ test("Type conflict in primitive field values") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldValueTypeConflict)
+
+ val expectedSchema =
+ AttributeReference("num_bool", StringType, true)() ::
+ AttributeReference("num_num_1", LongType, true)() ::
+ AttributeReference("num_num_2", DecimalType, true)() ::
+ AttributeReference("num_num_3", DoubleType, true)() ::
+ AttributeReference("num_str", StringType, true)() ::
+ AttributeReference("str_bool", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ ("true", 11L, null, 1.1, "13.1", "str1") ::
+ ("12", null, BigDecimal("21474836470.9"), null, null, "true") ::
+ ("false", 21474836470L, BigDecimal("92233720368547758070"), 100, "str1", "false") ::
+ (null, 21474836570L, BigDecimal(1.1), 21474836470L, "92233720368547758070", null) :: Nil
+ )
+
+ // Number and Boolean conflict: resolve the type as number in this query.
+ checkAnswer(
+ sql("select num_bool - 10 from jsonTable where num_bool > 11"),
+ 2
+ )
+
+ /*
+ // Right now, the analyzer does not insert a Cast for num_bool.
+ // Number and Boolean conflict: resolve the type as boolean in this query.
+ // TODO: Re-enable this test
+ checkAnswer(
+ sql("select num_bool from jsonTable where NOT num_bool"),
+ false
+ )
+ */
+
+ /*
+ // Right now, the analyzer does not know that num_bool should be treated as a boolean.
+ // Number and Boolean conflict: resolve the type as boolean in this query.
+ // TODO: Re-enable this test
+ checkAnswer(
+ sql("select num_bool from jsonTable where num_bool"),
+ true
+ )
+ */
+
+ // Widening to LongType
+ checkAnswer(
+ sql("select num_num_1 - 100 from jsonTable where num_num_1 > 11"),
+ Seq(21474836370L) :: Seq(21474836470L) :: Nil
+ )
+
+ checkAnswer(
+ sql("select num_num_1 - 100 from jsonTable where num_num_1 > 10"),
+ Seq(-89) :: Seq(21474836370L) :: Seq(21474836470L) :: Nil
+ )
+
+ // Widening to DecimalType
+ checkAnswer(
+ sql("select num_num_2 + 1.2 from jsonTable where num_num_2 > 1.1"),
+ Seq(BigDecimal("21474836472.1")) :: Seq(BigDecimal("92233720368547758071.2")) :: Nil
+ )
+
+ // Widening to DoubleType
+ checkAnswer(
+ sql("select num_num_3 + 1.2 from jsonTable where num_num_3 > 1.1"),
+ Seq(101.2) :: Seq(21474836471.2) :: Nil
+ )
+
+ /*
+ // Right now, we have a parsing error.
--- End diff --
Perhaps this query could be written with the DSL to avoid the parsing issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45637957
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45645165
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518342
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
--- End diff --
Actually - it might make sense to just get rid of the Option. If sampleRatio == 1.0, then it doesn't need to do sample.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518299
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
--- End diff --
i think u can move this to the previous line without wrapping it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13827230
--- Diff: docs/sql-programming-guide.md ---
@@ -98,7 +98,9 @@ sqlCtx = SQLContext(sc)
</div>
-## Running SQL on RDDs
+# Data Sources
--- End diff --
We should have something here. Maybe:
`Spark SQL supports operating of a variety of data sources though the SchemaRDD interface. Once a dataset has been loaded, it can be registered as a table and even joined with data from other sources.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46061921
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15773/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518290
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
--- End diff --
let's put this in a SchemaResolutionMode.scala ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46352148
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46389105
Thanks. I'm merging this in master & branch-1.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46372515
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15855/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45814384
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/999
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45645152
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520478
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
--- End diff --
Maybe we could move this into QueryTest or something instead of duplicating it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520544
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
+ case map: java.util.Map[String, Object] =>
+ // .map(identity) is used as a workaround of non-serializable Map
+ // generated by .mapValues.
+ // This issue is documented at https://issues.scala-lang.org/browse/SI-7005
+ map.toMap.mapValues(scalafy).map(identity)
+ case list: java.util.List[Object] =>
+ list.toList.map(scalafy)
+ case atom => atom
+ }
+
+ protected def parseJson(json: RDD[String]): RDD[Map[String, Any]] = {
+ // According to [Jackson-72: https://jira.codehaus.org/browse/JACKSON-72],
+ // ObjectMapper will not return BigDecimal when
+ // "DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" is disabled
+ // (see NumberDeserializer.deserialize for the logic).
+ // But, we do not want to enable this feature because it will use BigDecimal
+ // for every float number, which will be slow.
+ // So, right now, we will have Infinity for those BigDecimal number.
+ // TODO: Support BigDecimal.
+ json.mapPartitions(iter => {
+ // When there is a key appearing multiple times (a duplicate key),
+ // the ObjectMapper will take the last value associated with this duplicate key.
+ // For example: for {"key": 1, "key":2}, we will get "key"->2.
+ val mapper = new ObjectMapper()
+ iter.map(record => mapper.readValue(record, classOf[Object]))
+ }).map(scalafy).map(_.asInstanceOf[Map[String, Any]])
+ }
+
+ protected def toLong(value: Any): Long = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Long]
+ case value: java.lang.Long => value.asInstanceOf[Long]
+ }
+ }
+
+ protected def toDouble(value: Any): Double = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Double]
+ case value: java.lang.Long => value.asInstanceOf[Long].asInstanceOf[Double]
+ case value: java.lang.Double => value.asInstanceOf[Double]
+ }
+ }
+
+ protected def toDecimal(value: Any): BigDecimal = {
+ value match {
+ case value: java.lang.Integer => BigDecimal(value)
+ case value: java.lang.Long => BigDecimal(value)
+ case value: java.math.BigInteger => BigDecimal(value)
+ case value: java.lang.Double => BigDecimal(value)
+ case value: java.math.BigDecimal => BigDecimal(value)
+ }
+ }
+
+ protected def enforceCorrectType(value: Any, desiredType: DataType): Any ={
+ if (value == null) {
+ null
+ } else {
+ desiredType match {
+ case ArrayType(elementType) =>
+ value.asInstanceOf[Seq[Any]].map(enforceCorrectType(_, elementType))
+ case StringType => value.toString
+ case IntegerType => value.asInstanceOf[IntegerType.JvmType]
+ case LongType => toLong(value)
+ case DoubleType => toDouble(value)
+ case DecimalType => toDecimal(value)
+ case BooleanType => value.asInstanceOf[BooleanType.JvmType]
+ case NullType => null
+ }
+ }
+ }
+
+ protected def asRow(json: Map[String,Any], schema: StructType): Row = {
+ val row = new GenericMutableRow(schema.fields.length)
+ schema.fields.zipWithIndex.foreach {
+ // StructType
+ case (StructField(name, fields: StructType, _), i) =>
+ row.update(i, json.get(name).flatMap(v => Option(v)).map(
+ v => asRow(v.asInstanceOf[Map[String, Any]], fields)).orNull)
+
+ // ArrayType(StructType)
+ case (StructField(name, ArrayType(structType: StructType), _), i) =>
+ row.update(i,
+ json.get(name).flatMap(v => Option(v)).map(
+ v => v.asInstanceOf[Seq[Any]].map(
+ e => asRow(e.asInstanceOf[Map[String, Any]], structType))).orNull)
+
+ // Other cases
+ case (StructField(name, dataType, _), i) =>
+ row.update(i, json.get(name).flatMap(v => Option(v)).map(
+ enforceCorrectType(_, dataType)).getOrElse(null))
+ }
+
+ row
+ }
+
+ protected def asAttributes(struct: StructType): Seq[AttributeReference] = {
--- End diff --
Minor note: #360 adds library functions to catalyst for doing these sorts of conversions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46249650
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518281
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDDLike.scala ---
@@ -111,4 +111,10 @@ private[sql] trait SchemaRDDLike {
@Experimental
def saveAsTable(tableName: String): Unit =
sqlContext.executePlan(InsertIntoCreatedTable(None, tableName, logicalPlan)).toRdd
+
+
+ /**
+ * Print the schema of this SchemaRDD.
+ */
+ def printSchema = queryExecution.analyzed.printSchema()
--- End diff --
put the return type explicitly here (is it just Unit)?
We should explicitly define returns types of all public functions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45391188
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15518/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13839520
--- Diff: docs/sql-programming-guide.md ---
@@ -170,12 +191,11 @@ A schema can be applied to an existing RDD by calling `applySchema` and providin
for the JavaBean.
{% highlight java %}
-
-JavaSparkContext ctx = ...; // An existing JavaSparkContext.
-JavaSQLContext sqlCtx = new org.apache.spark.sql.api.java.JavaSQLContext(ctx)
+// sc is an existing JavaSparkContext.
+JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc)
// Load a text file and convert each line to a JavaBean.
-JavaRDD<Person> people = ctx.textFile("examples/src/main/resources/people.txt").map(
+JavaRDD<Person> people = sc.textFile("examples/src/main/resources/people.txt").map(
--- End diff --
I think you have changes here that are accidental (this actually reverts some improvements/fixes to this doc)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13830083
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDDLike.scala ---
@@ -122,4 +122,10 @@ private[sql] trait SchemaRDDLike {
@Experimental
def saveAsTable(tableName: String): Unit =
sqlContext.executePlan(InsertIntoCreatedTable(None, tableName, logicalPlan)).toRdd
+
+ /** Returns the output schema in the tree format. */
+ def getSchemaTreeString(): String = queryExecution.analyzed.getSchemaTreeString()
--- End diff --
no ().
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518308
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
--- End diff --
it would be more obvious what this does if you write it out, i.e.
```scala
val schemaData = if (sampleSchema.isDefined) json.sample(false, sampleSchema.get, 1) else json
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45417535
Yea definitely very exciting feature.
Also, should we consider getting rid of EAGER_SCHEMA_RESOLUTION and the case classes, and just have a schemaResolutionRatio. If it is 0.0, then it is purely lazy; if it is greater than 0.0, then ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520449
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
--- End diff --
A separate file for 4 lines of code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45386150
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520420
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
--- End diff --
We should definitely add a block level sample to Spark, though I wouldn't block this PR based on that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13830011
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala ---
@@ -123,4 +124,10 @@ abstract class QueryPlan[PlanType <: TreeNode[PlanType]] extends TreeNode[PlanTy
case other => Nil
}.toSeq
}
+
+ /** Returns the output schema in the tree format. */
+ def getSchemaTreeString(): String = plans.generateSchemaTreeString(output)
--- End diff --
and no () for purely functional methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518265
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/package.scala ---
@@ -17,8 +17,56 @@
package org.apache.spark.sql.catalyst
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.types.{StructField, DataType, ArrayType, StructType}
+
/**
* A a collection of common abstractions for query plans as well as
* a base logical plan representation.
*/
-package object plans
+package object plans {
+ def generateSchemaTreeString(schema: Seq[Attribute]): String = {
--- End diff --
mark this as private[sql]?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46249651
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15828/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13566856
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
--- End diff --
No, I think we will not extend this object. I was thinking to only expose `inferSchema` to users. Should we do that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46261327
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520487
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/TestJsonData.scala ---
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.test.TestSQLContext
+
+object TestJsonData {
+
+ val primitiveFieldAndType =
+ TestSQLContext.sparkContext.parallelize(
+ """{"string":"this is a simple string.", """ +
--- End diff --
""" lets you do multi line string literals, so I'd remove the +s
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828608
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -99,6 +97,35 @@ class SQLContext(@transient val sparkContext: SparkContext)
new SchemaRDD(this, parquet.ParquetRelation(path))
/**
+ * Loads a JSON file (one object per line), returning the result as a [[SchemaRDD]].
+ *
+ * @group userf
+ */
+ def jsonFile(path: String): SchemaRDD = jsonFile(path, 1.0)
+
+ /**
+ * :: Experimental ::
+ */
+ def jsonFile(path: String, samplingRatio: Double): SchemaRDD = {
+ val json = sparkContext.textFile(path)
+ jsonRDD(json, samplingRatio)
+ }
+
+ /**
+ * Loads a RDD[String] storing JSON objects (one object per record), returning the result as a
+ * [[SchemaRDD]].
+ *
+ * @group userf
+ */
+ def jsonRDD(json: RDD[String]): SchemaRDD = jsonRDD(json, 1.0)
+
+ /**
+ * :: Experimental ::
+ */
+ def jsonRDD(json: RDD[String], samplingRatio: Double): SchemaRDD =
--- End diff --
Annotation too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13840206
--- Diff: docs/sql-programming-guide.md ---
@@ -297,50 +328,152 @@ JavaSchemaRDD teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >=
<div data-lang="python" markdown="1">
{% highlight python %}
+# sqlContext from the previous example is used in this example.
-peopleTable # The SchemaRDD from the previous example.
+schemaPeople # The SchemaRDD from the previous example.
# SchemaRDDs can be saved as Parquet files, maintaining the schema information.
-peopleTable.saveAsParquetFile("people.parquet")
+schemaPeople.saveAsParquetFile("people.parquet")
# Read in the Parquet file created above. Parquet files are self-describing so the schema is preserved.
# The result of loading a parquet file is also a SchemaRDD.
-parquetFile = sqlCtx.parquetFile("people.parquet")
+parquetFile = sqlContext.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerAsTable("parquetFile");
-teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
-
+teenagers = sqlContext.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
+teenNames = teenagers.map(lambda p: "Name: " + p.name)
+for teenName in teenNames.collect():
+ print teenName
{% endhighlight %}
</div>
</div>
-## Writing Language-Integrated Relational Queries
+## JSON Datasets
+<div class="codetabs">
-**Language-Integrated queries are currently only supported in Scala.**
+<div data-lang="scala" markdown="1">
+Spark SQL can automatically infer the schema of a JSON dataset and load it as a SchemaRDD.
+This conversion can be done using one of two methods in a SQLContext:
-Spark SQL also supports a domain specific language for writing queries. Once again,
-using the data from the above examples:
+* `jsonFile` - loads data from a directory of JSON files where each line of the files is a JSON object.
+* `jsonRdd` - loads data from an existing RDD where each element of the RDD is a string containing a JSON object.
{% highlight scala %}
+// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
-import sqlContext._
-val people: RDD[Person] = ... // An RDD of case class objects, from the first example.
-// The following is the same as 'SELECT name FROM people WHERE age >= 10 AND age <= 19'
-val teenagers = people.where('age >= 10).where('age <= 19).select('name)
+// A JSON dataset is pointed by path.
--- End diff --
"is pointed to by path"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45382260
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15515/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828571
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -99,6 +97,35 @@ class SQLContext(@transient val sparkContext: SparkContext)
new SchemaRDD(this, parquet.ParquetRelation(path))
/**
+ * Loads a JSON file (one object per line), returning the result as a [[SchemaRDD]].
+ *
+ * @group userf
+ */
+ def jsonFile(path: String): SchemaRDD = jsonFile(path, 1.0)
+
+ /**
+ * :: Experimental ::
+ */
--- End diff --
Needs the `@experimental` annotation too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518341
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
--- End diff --
why do we mark these as protected? the outer level is an object - are you going to extend the object?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45645298
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45560876
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46249499
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13603527
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
+ val minId = attributes.map(_.exprId.id).min
+ attributes.map {
+ case a: AttributeReference =>
+ AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
+ }
+ }
+
+ protected def checkSchema(expected: Seq[Attribute], actual: Seq[Attribute]): Unit = {
+ val normalizedExpected = normalizeExprIds(expected).toSeq
+ val normalizedActual = normalizeExprIds(actual).toSeq
+ if (normalizedExpected != normalizedActual) {
+ fail(
+ s"""
+ |=== FAIL: Schemas do not match ===
+ |${sideBySide(
+ s"== Expected Schema ==\n" +
+ generateSchemaTreeString(normalizedExpected),
+ s"== Actual Schema ==\n" +
+ generateSchemaTreeString(normalizedActual)).mkString("\n")}
+ """.stripMargin)
+ }
+ }
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
+ // TODO: Re-enable the following test.
+ // Getting all values of a specific field from an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable"),
+ (Seq(true, false), Seq("str1", null)) :: Nil
+ )
+ */
+
+ // Access a struct and fields inside of it.
+ checkAnswer(
+ sql("select struct, struct.field1, struct.field2 from jsonTable"),
+ (
+ Seq(true, BigDecimal("92233720368547758070")),
+ true,
+ BigDecimal("92233720368547758070")) :: Nil
+ )
+
+ // Access an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1, structWithArrayFields.field2 from jsonTable"),
+ (Seq(4, 5, 6), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1[1], structWithArrayFields.field2[3] from jsonTable"),
+ (5, null) :: Nil
+ )
+ }
+
+ test("Type conflict in primitive field values") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldValueTypeConflict)
+
+ val expectedSchema =
+ AttributeReference("num_bool", StringType, true)() ::
+ AttributeReference("num_num_1", LongType, true)() ::
+ AttributeReference("num_num_2", DecimalType, true)() ::
+ AttributeReference("num_num_3", DoubleType, true)() ::
+ AttributeReference("num_str", StringType, true)() ::
+ AttributeReference("str_bool", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ ("true", 11L, null, 1.1, "13.1", "str1") ::
+ ("12", null, BigDecimal("21474836470.9"), null, null, "true") ::
+ ("false", 21474836470L, BigDecimal("92233720368547758070"), 100, "str1", "false") ::
+ (null, 21474836570L, BigDecimal(1.1), 21474836470L, "92233720368547758070", null) :: Nil
+ )
+
+ // Number and Boolean conflict: resolve the type as number in this query.
+ checkAnswer(
+ sql("select num_bool - 10 from jsonTable where num_bool > 11"),
+ 2
+ )
+
+ /*
+ // Right now, the analyzer does not insert a Cast for num_bool.
+ // Number and Boolean conflict: resolve the type as boolean in this query.
+ // TODO: Re-enable this test
+ checkAnswer(
+ sql("select num_bool from jsonTable where NOT num_bool"),
+ false
+ )
+ */
+
+ /*
+ // Right now, the analyzer does not know that num_bool should be treated as a boolean.
+ // Number and Boolean conflict: resolve the type as boolean in this query.
+ // TODO: Re-enable this test
+ checkAnswer(
+ sql("select num_bool from jsonTable where num_bool"),
+ true
+ )
+ */
+
+ // Widening to LongType
+ checkAnswer(
+ sql("select num_num_1 - 100 from jsonTable where num_num_1 > 11"),
+ Seq(21474836370L) :: Seq(21474836470L) :: Nil
+ )
+
+ checkAnswer(
+ sql("select num_num_1 - 100 from jsonTable where num_num_1 > 10"),
+ Seq(-89) :: Seq(21474836370L) :: Seq(21474836470L) :: Nil
+ )
+
+ // Widening to DecimalType
+ checkAnswer(
+ sql("select num_num_2 + 1.2 from jsonTable where num_num_2 > 1.1"),
+ Seq(BigDecimal("21474836472.1")) :: Seq(BigDecimal("92233720368547758071.2")) :: Nil
+ )
+
+ // Widening to DoubleType
+ checkAnswer(
+ sql("select num_num_3 + 1.2 from jsonTable where num_num_3 > 1.1"),
+ Seq(101.2) :: Seq(21474836471.2) :: Nil
+ )
+
+ /*
+ // Right now, we have a parsing error.
--- End diff --
I tried
```
jsonSchemaRDD.
where('num_str > BigDecimal("92233720368547758060")).
select('num_str + 1.2 as Symbol("num"))
```
But, the plan I got is
```
Project [(CAST(num_str#65:4, DoubleType) + 1.2) AS num#78]
Filter (CAST(CAST(num_str#65:4, DoubleType), DecimalType) > 92233720368547758060)
ExistingRdd [num_bool#61,num_num_1#62L,num_num_2#63,num_num_3#64,num_str#65,str_bool#66]
```
Seems we need to take a look at `HiveTypeCoercion`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45640690
My thought on `SchemaResolutionMode` is that we may be able to hide implementation details on how we infer the schema by using `SchemaResolutionMode` in `jsonFile` and `jsonRDD`. If we expose `samplingRatio` in `jsonFile` and `jsonRDD`, when we have other methods to infer the schema (lazy resolution), we need to change existing interfaces or add new interfaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46222714
Have made a few changes:
* Removed the special SchemaRDD (JsonRDD) for JSON datasets. Now, when users call `jsonFile` and `jsonRDD`, a SchemaRDD is returned.
* Added Java and Python APIs. For the Python API, SchemaRDD.javaToPython can also handle StructType now.
* Updated programming guide.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518243
--- Diff: project/SparkBuild.scala ---
@@ -486,9 +486,23 @@ object SparkBuild extends Build {
def sqlCoreSettings = sharedSettings ++ Seq(
name := "spark-sql",
libraryDependencies ++= Seq(
- "com.twitter" % "parquet-column" % parquetVersion,
- "com.twitter" % "parquet-hadoop" % parquetVersion
- )
+ "com.twitter" % "parquet-column" % parquetVersion,
+ "com.twitter" % "parquet-hadoop" % parquetVersion,
+ "com.fasterxml.jackson.core" % "jackson-core" % "2.3.2"
+ ),
+ initialCommands in console :=
+ """
+ |import org.apache.spark.sql.catalyst.analysis._
--- End diff --
do we need to import all of these?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13690133
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,485 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import scala.collection.mutable.HashSet
+
+import org.apache.spark.sql.catalyst.analysis.Star
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.json.JsonTable.enforceCorrectType
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext
+import org.apache.spark.sql.test.TestSQLContext._
+
+protected case class Schema(output: Seq[Attribute]) extends LeafNode
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ comparePlans(Schema(expectedSchema), Schema(jsonSchemaRDD.logicalPlan.output))
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ comparePlans(Schema(expectedSchema), Schema(jsonSchemaRDD.logicalPlan.output))
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
+ // TODO: Re-enable the following test.
+ // Getting all values of a specific field from an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable"),
+ (Seq(true, false), Seq("str1", null)) :: Nil
+ )
+ */
+
+ // Access a struct and fields inside of it.
+ checkAnswer(
+ sql("select struct, struct.field1, struct.field2 from jsonTable"),
+ (
+ Seq(true, BigDecimal("92233720368547758070")),
+ true,
+ BigDecimal("92233720368547758070")) :: Nil
+ )
+
+ // Access an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1, structWithArrayFields.field2 from jsonTable"),
+ (Seq(4, 5, 6), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1[1], structWithArrayFields.field2[3] from jsonTable"),
+ (5, null) :: Nil
+ )
+ }
+
+ test("Type conflict in primitive field values") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldValueTypeConflict)
+
+ val expectedSchema =
+ AttributeReference("num_bool", StringType, true)() ::
+ AttributeReference("num_num_1", LongType, true)() ::
+ AttributeReference("num_num_2", DecimalType, true)() ::
+ AttributeReference("num_num_3", DoubleType, true)() ::
+ AttributeReference("num_str", StringType, true)() ::
+ AttributeReference("str_bool", StringType, true)() :: Nil
+
+ comparePlans(Schema(expectedSchema), Schema(jsonSchemaRDD.logicalPlan.output))
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ ("true", 11L, null, 1.1, "13.1", "str1") ::
+ ("12", null, BigDecimal("21474836470.9"), null, null, "true") ::
+ ("false", 21474836470L, BigDecimal("92233720368547758070"), 100, "str1", "false") ::
+ (null, 21474836570L, BigDecimal(1.1), 21474836470L, "92233720368547758070", null) :: Nil
+ )
+
+ // Number and Boolean conflict: resolve the type as number in this query.
+ checkAnswer(
+ sql("select num_bool - 10 from jsonTable where num_bool > 11"),
+ 2
+ )
+
+ /*
+ // Right now, the analyzer does not promote strings in a boolean expreesion.
+ // Number and Boolean conflict: resolve the type as boolean in this query.
+ // TODO: Re-enable this test
+ checkAnswer(
--- End diff --
Instead of commenting this out, wrap it in `ignore(...) { }`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520469
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
--- End diff --
Should this still be commented out?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518303
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
--- End diff --
why does this need to be serializable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13826671
--- Diff: docs/sql-programming-guide.md ---
@@ -62,10 +62,10 @@ descendants. To create a basic SQLContext, all you need is a SparkContext.
{% highlight scala %}
val sc: SparkContext // An existing SparkContext.
-val sqlContext = new org.apache.spark.sql.SQLContext(sc)
+val sqlCtx = new org.apache.spark.sql.SQLContext(sc)
--- End diff --
I don't think it's generally good Scala style to shorten things unnecessarily (Context -> ctx).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46061920
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13893161
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala ---
@@ -342,13 +344,34 @@ class SchemaRDD(
def toJavaSchemaRDD: JavaSchemaRDD = new JavaSchemaRDD(sqlContext, logicalPlan)
private[sql] def javaToPython: JavaRDD[Array[Byte]] = {
--- End diff --
add some inline doc explaining this is used for the Python API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13827707
--- Diff: docs/sql-programming-guide.md ---
@@ -310,37 +325,190 @@ parquetFile = sqlCtx.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerAsTable("parquetFile");
teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
-
+teenNames = teenagers.map(lambda p: "Name: " + p.name)
+for teenName in teenNames.collect():
+ print teenName
{% endhighlight %}
</div>
</div>
-## Writing Language-Integrated Relational Queries
+## JSON Datasets
+<div class="codetabs">
+
+<div data-lang="scala" markdown="1">
+Spark SQL supports querying JSON datasets. To query a JSON dataset, a SchemaRDD needs to be created for this JSON dataset. There are two ways to create a SchemaRDD for a JSON dataset:
-**Language-Integrated queries are currently only supported in Scala.**
+1. Creating the SchemaRDD from text files that store one JSON object per line.
+2. Creating the SchemaRDD from a RDD of strings (`RDD[String]`) that stores one JSON object.
-Spark SQL also supports a domain specific language for writing queries. Once again,
-using the data from the above examples:
+The schema of a JSON dataset is automatically inferred when the SchemaRDD is created.
{% highlight scala %}
-val sqlContext = new org.apache.spark.sql.SQLContext(sc)
-import sqlContext._
-val people: RDD[Person] = ... // An RDD of case class objects, from the first example.
+// sc is an existing SparkContext.
+val sqlCtx = new org.apache.spark.sql.SQLContext(sc)
+
+// A JSON dataset is pointed by path.
+// The path can be either a single text file or a directory storing text files.
+val path = "examples/src/main/resources/people.json"
+// Create a SchemaRDD from the file(s) pointed by path
+val people = sqlCtx.jsonFile(path)
+
+// Because the schema of a JSON dataset is automatically inferred, to write queries,
--- End diff --
"The inferred schema can be visualized using the `printSchema()` method."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13891482
--- Diff: docs/sql-programming-guide.md ---
@@ -91,14 +91,33 @@ of its decedents. To create a basic SQLContext, all you need is a SparkContext.
{% highlight python %}
from pyspark.sql import SQLContext
-sqlCtx = SQLContext(sc)
+sqlContext = SQLContext(sc)
{% endhighlight %}
</div>
</div>
-## Running SQL on RDDs
+# Data Sources
+
+<div class="codetabs">
+<div data-lang="scala" markdown="1">
+Spark SQL supports operating on a variety of data sources though the SchemaRDD interface.
--- End diff --
and for Python/Java too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13891473
--- Diff: docs/sql-programming-guide.md ---
@@ -91,14 +91,33 @@ of its decedents. To create a basic SQLContext, all you need is a SparkContext.
{% highlight python %}
from pyspark.sql import SQLContext
-sqlCtx = SQLContext(sc)
+sqlContext = SQLContext(sc)
{% endhighlight %}
</div>
</div>
-## Running SQL on RDDs
+# Data Sources
+
+<div class="codetabs">
+<div data-lang="scala" markdown="1">
+Spark SQL supports operating on a variety of data sources though the SchemaRDD interface.
--- End diff --
best to put <code> ... </code> around SchemaRDD
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520486
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
+ val minId = attributes.map(_.exprId.id).min
+ attributes.map {
+ case a: AttributeReference =>
+ AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
+ }
+ }
+
+ protected def checkSchema(expected: Seq[Attribute], actual: Seq[Attribute]): Unit = {
+ val normalizedExpected = normalizeExprIds(expected).toSeq
+ val normalizedActual = normalizeExprIds(actual).toSeq
+ if (normalizedExpected != normalizedActual) {
+ fail(
+ s"""
+ |=== FAIL: Schemas do not match ===
+ |${sideBySide(
+ s"== Expected Schema ==\n" +
+ generateSchemaTreeString(normalizedExpected),
+ s"== Actual Schema ==\n" +
+ generateSchemaTreeString(normalizedActual)).mkString("\n")}
+ """.stripMargin)
+ }
+ }
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
+ // TODO: Re-enable the following test.
+ // Getting all values of a specific field from an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable"),
+ (Seq(true, false), Seq("str1", null)) :: Nil
+ )
+ */
+
+ // Access a struct and fields inside of it.
+ checkAnswer(
+ sql("select struct, struct.field1, struct.field2 from jsonTable"),
+ (
+ Seq(true, BigDecimal("92233720368547758070")),
+ true,
+ BigDecimal("92233720368547758070")) :: Nil
+ )
+
+ // Access an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1, structWithArrayFields.field2 from jsonTable"),
+ (Seq(4, 5, 6), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1[1], structWithArrayFields.field2[3] from jsonTable"),
+ (5, null) :: Nil
+ )
+ }
+
+ test("Type conflict in primitive field values") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldValueTypeConflict)
+
+ val expectedSchema =
+ AttributeReference("num_bool", StringType, true)() ::
+ AttributeReference("num_num_1", LongType, true)() ::
+ AttributeReference("num_num_2", DecimalType, true)() ::
+ AttributeReference("num_num_3", DoubleType, true)() ::
+ AttributeReference("num_str", StringType, true)() ::
+ AttributeReference("str_bool", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ ("true", 11L, null, 1.1, "13.1", "str1") ::
+ ("12", null, BigDecimal("21474836470.9"), null, null, "true") ::
+ ("false", 21474836470L, BigDecimal("92233720368547758070"), 100, "str1", "false") ::
+ (null, 21474836570L, BigDecimal(1.1), 21474836470L, "92233720368547758070", null) :: Nil
+ )
+
+ // Number and Boolean conflict: resolve the type as number in this query.
+ checkAnswer(
+ sql("select num_bool - 10 from jsonTable where num_bool > 11"),
+ 2
+ )
+
+ /*
+ // Right now, the analyzer does not insert a Cast for num_bool.
--- End diff --
What do you mean here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518251
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala ---
@@ -108,6 +108,9 @@ trait HiveTypeCoercion {
*
* Additionally, all types when UNION-ed with strings will be promoted to strings.
* Other string conversions are handled by PromoteStrings.
+ *
+ * A widening conversion of a value with IntegerType and LongType to FloatType,
--- End diff --
Maybe more clear if you document it this way:
```
Widening types might result in loss of precision the following cases:
- IntegerType to FloatType
- LongType to FloatType
- LongType to DoubleType
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518284
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
--- End diff --
order the imports according to https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide#SparkCodeStyleGuide-Imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828440
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerTest.scala ---
@@ -17,39 +17,10 @@
package org.apache.spark.sql.catalyst.optimizer
-import org.scalatest.FunSuite
-
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.util._
+import org.apache.spark.sql.catalyst.plans.PlanTest
/**
* Provides helper methods for comparing plans produced by optimization rules with the expected
* result
*/
-class OptimizerTest extends FunSuite {
-
- /**
- * Since attribute references are given globally unique ids during analysis,
- * we must normalize them to check if two different queries are identical.
- */
- protected def normalizeExprIds(plan: LogicalPlan) = {
- val minId = plan.flatMap(_.expressions.flatMap(_.references).map(_.exprId.id)).min
- plan transformAllExpressions {
- case a: AttributeReference =>
- AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
- }
- }
-
- /** Fails the test if the two plans do not match */
- protected def comparePlans(plan1: LogicalPlan, plan2: LogicalPlan) {
- val normalized1 = normalizeExprIds(plan1)
- val normalized2 = normalizeExprIds(plan2)
- if (normalized1 != normalized2)
- fail(
- s"""
- |== FAIL: Plans do not match ===
- |${sideBySide(normalized1.treeString, normalized2.treeString).mkString("\n")}
- """.stripMargin)
- }
-}
+class OptimizerTest extends PlanTest
--- End diff --
Is there any reason to keep this class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45643411
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45391187
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13829492
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDDLike.scala ---
@@ -122,4 +122,10 @@ private[sql] trait SchemaRDDLike {
@Experimental
def saveAsTable(tableName: String): Unit =
sqlContext.executePlan(InsertIntoCreatedTable(None, tableName, logicalPlan)).toRdd
+
+ /** Returns the output schema in the tree format. */
+ def getSchemaTreeString(): String = queryExecution.analyzed.getSchemaTreeString()
--- End diff --
No `get`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45386159
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520414
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
--- End diff --
It was because I started writing this in the REPL and it kept getting pulled into jobs for no good reason. Its possible that we could remove it now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13829475
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala ---
@@ -123,4 +124,10 @@ abstract class QueryPlan[PlanType <: TreeNode[PlanType]] extends TreeNode[PlanTy
case other => Nil
}.toSeq
}
+
+ /** Returns the output schema in the tree format. */
+ def getSchemaTreeString(): String = plans.generateSchemaTreeString(output)
--- End diff --
Don't use `get` for scala functions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13827260
--- Diff: docs/sql-programming-guide.md ---
@@ -98,7 +98,9 @@ sqlCtx = SQLContext(sc)
</div>
-## Running SQL on RDDs
+# Data Sources
+
+## RDDs
--- End diff --
These second level headings don't appear to be formatting any different than first level ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46046141
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46046033
Programming guide: http://yhuai.github.io/site/sql-programming-guide.html
Scala doc of SQLContext: http://yhuai.github.io/site/api/scala/index.html#org.apache.spark.sql.SQLContext
Scala doc of JsonTable: http://yhuai.github.io/site/api/scala/index.html#org.apache.spark.sql.json.JsonTable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518348
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
+ case map: java.util.Map[String, Object] =>
+ // .map(identity) is used as a workaround of non-serializable Map
+ // generated by .mapValues.
+ // This issue is documented at https://issues.scala-lang.org/browse/SI-7005
+ map.toMap.mapValues(scalafy).map(identity)
+ case list: java.util.List[Object] =>
+ list.toList.map(scalafy)
+ case atom => atom
+ }
+
+ protected def parseJson(json: RDD[String]): RDD[Map[String, Any]] = {
+ // According to [Jackson-72: https://jira.codehaus.org/browse/JACKSON-72],
+ // ObjectMapper will not return BigDecimal when
+ // "DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" is disabled
+ // (see NumberDeserializer.deserialize for the logic).
+ // But, we do not want to enable this feature because it will use BigDecimal
+ // for every float number, which will be slow.
+ // So, right now, we will have Infinity for those BigDecimal number.
+ // TODO: Support BigDecimal.
+ json.mapPartitions(iter => {
+ // When there is a key appearing multiple times (a duplicate key),
+ // the ObjectMapper will take the last value associated with this duplicate key.
+ // For example: for {"key": 1, "key":2}, we will get "key"->2.
+ val mapper = new ObjectMapper()
+ iter.map(record => mapper.readValue(record, classOf[Object]))
+ }).map(scalafy).map(_.asInstanceOf[Map[String, Any]])
+ }
+
+ protected def toLong(value: Any): Long = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Long]
--- End diff --
value.asInstanceOf[Int].toLong
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13601226
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
--- End diff --
We will use functions provided by Scala collections (e.g. map and flatMap). So, we explicitly convert the type from Java Map/Lists to Scala Map/List.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13839460
--- Diff: docs/sql-programming-guide.md ---
@@ -91,14 +91,33 @@ of its decedents. To create a basic SQLContext, all you need is a SparkContext.
{% highlight python %}
from pyspark.sql import SQLContext
-sqlCtx = SQLContext(sc)
+sqlContext = SQLContext(sc)
{% endhighlight %}
</div>
</div>
-## Running SQL on RDDs
+# Data Sources
+
+<div class="codetabs">
+<div data-lang="scala" markdown="1">
+Spark SQL supports operating of a variety of data sources though the SchemaRDD interface.
--- End diff --
This should be "operating on a variety"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45383097
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520483
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
+ val minId = attributes.map(_.exprId.id).min
+ attributes.map {
+ case a: AttributeReference =>
+ AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
+ }
+ }
+
+ protected def checkSchema(expected: Seq[Attribute], actual: Seq[Attribute]): Unit = {
+ val normalizedExpected = normalizeExprIds(expected).toSeq
+ val normalizedActual = normalizeExprIds(actual).toSeq
+ if (normalizedExpected != normalizedActual) {
+ fail(
+ s"""
+ |=== FAIL: Schemas do not match ===
+ |${sideBySide(
+ s"== Expected Schema ==\n" +
+ generateSchemaTreeString(normalizedExpected),
+ s"== Actual Schema ==\n" +
+ generateSchemaTreeString(normalizedActual)).mkString("\n")}
+ """.stripMargin)
+ }
+ }
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
--- End diff --
Create a JIRA for this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45638490
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518330
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
--- End diff --
what's wrong with the java map/lists?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45383282
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15516/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46360653
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13834977
--- Diff: docs/sql-programming-guide.md ---
@@ -310,37 +325,190 @@ parquetFile = sqlCtx.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerAsTable("parquetFile");
teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
-
+teenNames = teenagers.map(lambda p: "Name: " + p.name)
+for teenName in teenNames.collect():
+ print teenName
{% endhighlight %}
</div>
</div>
-## Writing Language-Integrated Relational Queries
+## JSON Datasets
+<div class="codetabs">
+
+<div data-lang="scala" markdown="1">
+Spark SQL supports querying JSON datasets. To query a JSON dataset, a SchemaRDD needs to be created for this JSON dataset. There are two ways to create a SchemaRDD for a JSON dataset:
--- End diff --
How about we add the section on the rules of dealing schema inference in the work on type system?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46405318
Actually we were explicit about alpha component updates in the discussion about versioning. One requirement though is the update to the alpha components shouldn't affect the non-alpha parts. That's why I asked @yhuai to actually avoid updating jackson version in this pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45416400
I'm excited to merge this in :)
One other thing: it would be nice to build the updated scala doc and publish them somewhere to make it easier to understand what we are adding to the public API
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45645300
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15629/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13629403
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
+ case map: java.util.Map[String, Object] =>
+ // .map(identity) is used as a workaround of non-serializable Map
+ // generated by .mapValues.
+ // This issue is documented at https://issues.scala-lang.org/browse/SI-7005
+ map.toMap.mapValues(scalafy).map(identity)
+ case list: java.util.List[Object] =>
+ list.toList.map(scalafy)
+ case atom => atom
+ }
+
+ protected def parseJson(json: RDD[String]): RDD[Map[String, Any]] = {
+ // According to [Jackson-72: https://jira.codehaus.org/browse/JACKSON-72],
+ // ObjectMapper will not return BigDecimal when
+ // "DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" is disabled
+ // (see NumberDeserializer.deserialize for the logic).
+ // But, we do not want to enable this feature because it will use BigDecimal
+ // for every float number, which will be slow.
+ // So, right now, we will have Infinity for those BigDecimal number.
+ // TODO: Support BigDecimal.
+ json.mapPartitions(iter => {
+ // When there is a key appearing multiple times (a duplicate key),
+ // the ObjectMapper will take the last value associated with this duplicate key.
+ // For example: for {"key": 1, "key":2}, we will get "key"->2.
+ val mapper = new ObjectMapper()
+ iter.map(record => mapper.readValue(record, classOf[Object]))
+ }).map(scalafy).map(_.asInstanceOf[Map[String, Any]])
+ }
+
+ protected def toLong(value: Any): Long = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Long]
+ case value: java.lang.Long => value.asInstanceOf[Long]
+ }
+ }
+
+ protected def toDouble(value: Any): Double = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Double]
+ case value: java.lang.Long => value.asInstanceOf[Long].asInstanceOf[Double]
+ case value: java.lang.Double => value.asInstanceOf[Double]
+ }
+ }
+
+ protected def toDecimal(value: Any): BigDecimal = {
+ value match {
+ case value: java.lang.Integer => BigDecimal(value)
+ case value: java.lang.Long => BigDecimal(value)
+ case value: java.math.BigInteger => BigDecimal(value)
+ case value: java.lang.Double => BigDecimal(value)
+ case value: java.math.BigDecimal => BigDecimal(value)
+ }
+ }
+
+ protected def enforceCorrectType(value: Any, desiredType: DataType): Any ={
+ if (value == null) {
+ null
+ } else {
+ desiredType match {
+ case ArrayType(elementType) =>
+ value.asInstanceOf[Seq[Any]].map(enforceCorrectType(_, elementType))
+ case StringType => value.toString
+ case IntegerType => value.asInstanceOf[IntegerType.JvmType]
--- End diff --
I need to change this part. So, we can automatically update the type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518292
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
--- End diff --
u don't need the val for case class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13840531
--- Diff: docs/sql-programming-guide.md ---
@@ -170,12 +191,11 @@ A schema can be applied to an existing RDD by calling `applySchema` and providin
for the JavaBean.
{% highlight java %}
-
-JavaSparkContext ctx = ...; // An existing JavaSparkContext.
-JavaSQLContext sqlCtx = new org.apache.spark.sql.api.java.JavaSQLContext(ctx)
+// sc is an existing JavaSparkContext.
+JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc)
// Load a text file and convert each line to a JavaBean.
-JavaRDD<Person> people = ctx.textFile("examples/src/main/resources/people.txt").map(
+JavaRDD<Person> people = sc.textFile("examples/src/main/resources/people.txt").map(
--- End diff --
yeah, I was trying to make variable names consistent for these three languages. If there is any convention for a specific language, I can revert my changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520397
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/package.scala ---
@@ -17,8 +17,56 @@
package org.apache.spark.sql.catalyst
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.types.{StructField, DataType, ArrayType, StructType}
+
/**
* A a collection of common abstractions for query plans as well as
* a base logical plan representation.
*/
-package object plans
+package object plans {
+ def generateSchemaTreeString(schema: Seq[Attribute]): String = {
--- End diff --
We don't mark any of catalyst as private. We just don't include any of the scala doc in the default spark distribution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13522696
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
--- End diff --
Be careful with floating point comparison though -- `0.3*3 + .1 != 1.0` for example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46061886
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518326
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
--- End diff --
I think you can put this whole thing on one line too...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13826595
--- Diff: docs/sql-programming-guide.md ---
@@ -17,20 +17,20 @@ Spark. At the core of this component is a new type of RDD,
[Row](api/scala/index.html#org.apache.spark.sql.catalyst.expressions.Row) objects along with
a schema that describes the data types of each column in the row. A SchemaRDD is similar to a table
in a traditional relational database. A SchemaRDD can be created from an existing RDD, [Parquet](http://parquet.io)
-file, or by running HiveQL against data stored in [Apache Hive](http://hive.apache.org/).
+file, a JSON dataset, or by running HiveQL against data stored in [Apache Hive](http://hive.apache.org/).
All of the examples on this page use sample data included in the Spark distribution and can be run in the `spark-shell`.
</div>
<div data-lang="java" markdown="1">
-Spark SQL allows relational queries expressed in SQL, HiveQL, or Scala to be executed using
+Spark SQL allows relational queries expressed in SQL or HiveQL to be executed using
--- End diff --
Why remove scala?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46222401
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15821/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518263
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/package.scala ---
@@ -17,8 +17,56 @@
package org.apache.spark.sql.catalyst
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.types.{StructField, DataType, ArrayType, StructType}
+
/**
* A a collection of common abstractions for query plans as well as
* a base logical plan representation.
*/
-package object plans
+package object plans {
+ def generateSchemaTreeString(schema: Seq[Attribute]): String = {
+ val builder = new StringBuilder
+ builder.append("root\n")
+ val prefix = " |"
+ schema.foreach {
+ attribute => {
+ val name = attribute.name
+ val dataType = attribute.dataType
+ dataType match {
+ case fields: StructType =>
+ builder.append(s"$prefix-- $name: $StructType\n")
+ generateSchemaTreeString(fields, s"$prefix |", builder)
+ case ArrayType(fields: StructType) =>
+ builder.append(s"$prefix-- $name: $ArrayType[$StructType]\n")
+ generateSchemaTreeString(fields, s"$prefix |", builder)
+ case ArrayType(elementType: DataType) =>
+ builder.append(s"$prefix-- $name: $ArrayType[$elementType]\n")
+ case _ => builder.append(s"$prefix-- $name: $dataType\n")
+ }
+ }
+ }
+
+ builder.toString()
+ }
+
+ def generateSchemaTreeString(
--- End diff --
mark this private?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828940
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/api/java/JavaSQLSuite.scala ---
@@ -17,11 +17,13 @@
package org.apache.spark.sql.api.java
+
--- End diff --
Extra line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46264603
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46222342
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46111039
@mateiz, thats a good point and actually there is only a single implicit conversion needed for all the non-DSL examples (from RDD -> SchemaRDD). Perhaps we could import that explicitly and move the other expression conversions to a dsl object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13892874
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -99,6 +97,37 @@ class SQLContext(@transient val sparkContext: SparkContext)
new SchemaRDD(this, parquet.ParquetRelation(path))
/**
+ * Loads a JSON file (one object per line), returning the result as a [[SchemaRDD]].
--- End diff --
Maybe add a line explaining this goes through the data once to infer the schema ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46529253
Ah yeah, this might not have been super clear, but this has at least been my assumption. We do want to make both SQL and GraphX be non-alpha soon though, perhaps as early as 1.1. GraphX is closer but SQL has such a narrow external API that I think it's good to lock it down.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520443
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -98,6 +99,41 @@ class SQLContext(@transient val sparkContext: SparkContext)
new SchemaRDD(this, parquet.ParquetRelation(path))
/**
+ * Loads a JSON file, returning the result as a [[SchemaRDD]].
+ * Right now, we only do eager schema resolution.
+ */
+ def jsonFile(
+ path: String,
+ mode: SchemaResolutionMode = EAGER_SCHEMA_RESOLUTION): SchemaRDD = {
+ logger.info(s"Loads a JSON file $path.")
+ val json = sparkContext.textFile(path)
+ jsonRDD(json, mode)
+ }
+
+ /**
+ * Loads a RDD[String] storing JSON objects (one object per record),
+ * returning the result as a [[SchemaRDD]].
+ * Right now, we only do eager schema resolution.
+ */
+ def jsonRDD(
+ json: RDD[String],
+ mode: SchemaResolutionMode = EAGER_SCHEMA_RESOLUTION): SchemaRDD = {
+ mode match {
+ case EAGER_SCHEMA_RESOLUTION =>
+ logger.info(s"Eagerly resolve the schema without sampling.")
+ val logicalPlan = JsonTable.inferSchema(json)
+ logicalPlanToSparkQuery(logicalPlan)
+ case EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(fraction) =>
+ logger.info(s"Eagerly resolve the schema with sampling " +
+ s"(sampling fraction: $fraction).")
+ val logicalPlan = JsonTable.inferSchema(json, Some(fraction))
+ logicalPlanToSparkQuery(logicalPlan)
+ case LAZY_SCHEMA_RESOLUTION =>
--- End diff --
Perhaps we should just leave this out so users will get compile errors instead of runtime errors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13783210
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDDLike.scala ---
@@ -122,4 +122,9 @@ private[sql] trait SchemaRDDLike {
@Experimental
def saveAsTable(tableName: String): Unit =
sqlContext.executePlan(InsertIntoCreatedTable(None, tableName, logicalPlan)).toRdd
+
+ /**
+ * Print the schema of this SchemaRDD.
+ */
+ def printSchema(): Unit = queryExecution.analyzed.printSchema()
--- End diff --
We should also add another method to return this as a string instead of printing it out
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46101559
Hey Yin, a few comments on the docs:
- You should mention JSON as a data source in the first paragraph of the Spark SQL doc (right now it only mentions Parquet and Hive)
- We may want to reorganize the doc to list all the external data sources in the same section. It's weird that Parquet and JSON are under "getting started", but then Hive is a separate section, and the DSL stuff is in-between. I'd actually put the DSL last and move the other there into a new top-level section called "data sources". You can also leave a quick example of one data source in Getting Started.
- There need to be Java and Python versions of the JSON example (I guess you're waiting to implement those APIs?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45643415
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15628/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45382259
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518282
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDDLike.scala ---
@@ -111,4 +111,10 @@ private[sql] trait SchemaRDDLike {
@Experimental
def saveAsTable(tableName: String): Unit =
sqlContext.executePlan(InsertIntoCreatedTable(None, tableName, logicalPlan)).toRdd
+
--- End diff --
remove the extra line here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13840123
--- Diff: docs/sql-programming-guide.md ---
@@ -170,12 +191,11 @@ A schema can be applied to an existing RDD by calling `applySchema` and providin
for the JavaBean.
{% highlight java %}
-
-JavaSparkContext ctx = ...; // An existing JavaSparkContext.
-JavaSQLContext sqlCtx = new org.apache.spark.sql.api.java.JavaSQLContext(ctx)
+// sc is an existing JavaSparkContext.
+JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc)
// Load a text file and convert each line to a JavaBean.
-JavaRDD<Person> people = ctx.textFile("examples/src/main/resources/people.txt").map(
+JavaRDD<Person> people = sc.textFile("examples/src/main/resources/people.txt").map(
--- End diff --
actually sorry maybe this is intentional?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46250558
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46363656
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46222400
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45814358
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45814385
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15694/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46061901
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518257
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/package.scala ---
@@ -17,8 +17,56 @@
package org.apache.spark.sql.catalyst
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.types.{StructField, DataType, ArrayType, StructType}
--- End diff --
order the imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518261
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/package.scala ---
@@ -17,8 +17,56 @@
package org.apache.spark.sql.catalyst
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.types.{StructField, DataType, ArrayType, StructType}
+
/**
* A a collection of common abstractions for query plans as well as
* a base logical plan representation.
*/
-package object plans
+package object plans {
+ def generateSchemaTreeString(schema: Seq[Attribute]): String = {
+ val builder = new StringBuilder
+ builder.append("root\n")
+ val prefix = " |"
+ schema.foreach {
+ attribute => {
--- End diff --
put attribute on the previous line to reduce indenting, i.e.
```scala
schema.foreach { attribute =>
}
```
(also reduced one level of curly braces)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46046123
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45642845
API doc for sql/core http://yhuai.github.io/spark-sql-core/api/#package
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45568509
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46255971
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13891733
--- Diff: docs/sql-programming-guide.md ---
@@ -297,50 +328,152 @@ JavaSchemaRDD teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >=
<div data-lang="python" markdown="1">
{% highlight python %}
+# sqlContext from the previous example is used in this example.
-peopleTable # The SchemaRDD from the previous example.
+schemaPeople # The SchemaRDD from the previous example.
# SchemaRDDs can be saved as Parquet files, maintaining the schema information.
-peopleTable.saveAsParquetFile("people.parquet")
+schemaPeople.saveAsParquetFile("people.parquet")
# Read in the Parquet file created above. Parquet files are self-describing so the schema is preserved.
# The result of loading a parquet file is also a SchemaRDD.
-parquetFile = sqlCtx.parquetFile("people.parquet")
+parquetFile = sqlContext.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerAsTable("parquetFile");
-teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
-
+teenagers = sqlContext.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
+teenNames = teenagers.map(lambda p: "Name: " + p.name)
+for teenName in teenNames.collect():
+ print teenName
{% endhighlight %}
</div>
</div>
-## Writing Language-Integrated Relational Queries
+## JSON Datasets
+<div class="codetabs">
-**Language-Integrated queries are currently only supported in Scala.**
+<div data-lang="scala" markdown="1">
+Spark SQL can automatically infer the schema of a JSON dataset and load it as a SchemaRDD.
+This conversion can be done using one of two methods in a SQLContext:
-Spark SQL also supports a domain specific language for writing queries. Once again,
-using the data from the above examples:
+* `jsonFile` - loads data from a directory of JSON files where each line of the files is a JSON object.
+* `jsonRdd` - loads data from an existing RDD where each element of the RDD is a string containing a JSON object.
{% highlight scala %}
+// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
-import sqlContext._
-val people: RDD[Person] = ... // An RDD of case class objects, from the first example.
-// The following is the same as 'SELECT name FROM people WHERE age >= 10 AND age <= 19'
-val teenagers = people.where('age >= 10).where('age <= 19).select('name)
+// A JSON dataset is pointed to by path.
+// The path can be either a single text file or a directory storing text files.
+val path = "examples/src/main/resources/people.json"
+// Create a SchemaRDD from the file(s) pointed to by path
+val people = sqlContext.jsonFile(path)
+
+// The inferred schema can be visualized using the printSchema() method.
+people.printSchema()
+// The schema of people is ...
--- End diff --
i'd remove this line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46046144
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15768/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828715
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala ---
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import org.apache.spark.annotation.{DeveloperApi, Experimental}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.Logging
+
+@Experimental
--- End diff --
We can remove these annotations if the object is not in the public API.
Also, some developer facing scala doc would be good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46047263
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15769/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520412
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
--- End diff --
Also we don't use ALL_CAPS for class names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45643143
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46261318
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13602167
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
+ val minId = attributes.map(_.exprId.id).min
+ attributes.map {
+ case a: AttributeReference =>
+ AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
+ }
+ }
+
+ protected def checkSchema(expected: Seq[Attribute], actual: Seq[Attribute]): Unit = {
+ val normalizedExpected = normalizeExprIds(expected).toSeq
+ val normalizedActual = normalizeExprIds(actual).toSeq
+ if (normalizedExpected != normalizedActual) {
+ fail(
+ s"""
+ |=== FAIL: Schemas do not match ===
+ |${sideBySide(
+ s"== Expected Schema ==\n" +
+ generateSchemaTreeString(normalizedExpected),
+ s"== Actual Schema ==\n" +
+ generateSchemaTreeString(normalizedActual)).mkString("\n")}
+ """.stripMargin)
+ }
+ }
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
+ // TODO: Re-enable the following test.
+ // Getting all values of a specific field from an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable"),
+ (Seq(true, false), Seq("str1", null)) :: Nil
+ )
+ */
+
+ // Access a struct and fields inside of it.
+ checkAnswer(
+ sql("select struct, struct.field1, struct.field2 from jsonTable"),
+ (
+ Seq(true, BigDecimal("92233720368547758070")),
+ true,
+ BigDecimal("92233720368547758070")) :: Nil
+ )
+
+ // Access an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1, structWithArrayFields.field2 from jsonTable"),
+ (Seq(4, 5, 6), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array field of a struct.
+ checkAnswer(
+ sql("select structWithArrayFields.field1[1], structWithArrayFields.field2[3] from jsonTable"),
+ (5, null) :: Nil
+ )
+ }
+
+ test("Type conflict in primitive field values") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldValueTypeConflict)
+
+ val expectedSchema =
+ AttributeReference("num_bool", StringType, true)() ::
+ AttributeReference("num_num_1", LongType, true)() ::
+ AttributeReference("num_num_2", DecimalType, true)() ::
+ AttributeReference("num_num_3", DoubleType, true)() ::
+ AttributeReference("num_str", StringType, true)() ::
+ AttributeReference("str_bool", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ ("true", 11L, null, 1.1, "13.1", "str1") ::
+ ("12", null, BigDecimal("21474836470.9"), null, null, "true") ::
+ ("false", 21474836470L, BigDecimal("92233720368547758070"), 100, "str1", "false") ::
+ (null, 21474836570L, BigDecimal(1.1), 21474836470L, "92233720368547758070", null) :: Nil
+ )
+
+ // Number and Boolean conflict: resolve the type as number in this query.
+ checkAnswer(
+ sql("select num_bool - 10 from jsonTable where num_bool > 11"),
+ 2
+ )
+
+ /*
+ // Right now, the analyzer does not insert a Cast for num_bool.
--- End diff --
In the query, we have a `NOT num_bool`. But, the type of column `num_bool` is `StringType`. Seems the analyzer does not promote strings that appear in boolean expression.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13604518
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
+ case map: java.util.Map[String, Object] =>
+ // .map(identity) is used as a workaround of non-serializable Map
+ // generated by .mapValues.
+ // This issue is documented at https://issues.scala-lang.org/browse/SI-7005
+ map.toMap.mapValues(scalafy).map(identity)
+ case list: java.util.List[Object] =>
+ list.toList.map(scalafy)
+ case atom => atom
+ }
+
+ protected def parseJson(json: RDD[String]): RDD[Map[String, Any]] = {
+ // According to [Jackson-72: https://jira.codehaus.org/browse/JACKSON-72],
+ // ObjectMapper will not return BigDecimal when
+ // "DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" is disabled
+ // (see NumberDeserializer.deserialize for the logic).
+ // But, we do not want to enable this feature because it will use BigDecimal
+ // for every float number, which will be slow.
+ // So, right now, we will have Infinity for those BigDecimal number.
+ // TODO: Support BigDecimal.
+ json.mapPartitions(iter => {
+ // When there is a key appearing multiple times (a duplicate key),
+ // the ObjectMapper will take the last value associated with this duplicate key.
+ // For example: for {"key": 1, "key":2}, we will get "key"->2.
+ val mapper = new ObjectMapper()
+ iter.map(record => mapper.readValue(record, classOf[Object]))
+ }).map(scalafy).map(_.asInstanceOf[Map[String, Any]])
+ }
+
+ protected def toLong(value: Any): Long = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Long]
+ case value: java.lang.Long => value.asInstanceOf[Long]
+ }
+ }
+
+ protected def toDouble(value: Any): Double = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Double]
+ case value: java.lang.Long => value.asInstanceOf[Long].asInstanceOf[Double]
+ case value: java.lang.Double => value.asInstanceOf[Double]
+ }
+ }
+
+ protected def toDecimal(value: Any): BigDecimal = {
+ value match {
+ case value: java.lang.Integer => BigDecimal(value)
+ case value: java.lang.Long => BigDecimal(value)
+ case value: java.math.BigInteger => BigDecimal(value)
+ case value: java.lang.Double => BigDecimal(value)
+ case value: java.math.BigDecimal => BigDecimal(value)
+ }
+ }
+
+ protected def enforceCorrectType(value: Any, desiredType: DataType): Any ={
+ if (value == null) {
+ null
+ } else {
+ desiredType match {
+ case ArrayType(elementType) =>
+ value.asInstanceOf[Seq[Any]].map(enforceCorrectType(_, elementType))
+ case StringType => value.toString
+ case IntegerType => value.asInstanceOf[IntegerType.JvmType]
--- End diff --
For this part, I think an exception means that we inferred the wrong data type for a column.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46260432
I just did a really light pass on the docs and public interfaces exposed. From that perspective, this looks good to me!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520563
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
+ case map: java.util.Map[String, Object] =>
+ // .map(identity) is used as a workaround of non-serializable Map
+ // generated by .mapValues.
+ // This issue is documented at https://issues.scala-lang.org/browse/SI-7005
+ map.toMap.mapValues(scalafy).map(identity)
+ case list: java.util.List[Object] =>
+ list.toList.map(scalafy)
+ case atom => atom
+ }
+
+ protected def parseJson(json: RDD[String]): RDD[Map[String, Any]] = {
+ // According to [Jackson-72: https://jira.codehaus.org/browse/JACKSON-72],
+ // ObjectMapper will not return BigDecimal when
+ // "DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" is disabled
+ // (see NumberDeserializer.deserialize for the logic).
+ // But, we do not want to enable this feature because it will use BigDecimal
+ // for every float number, which will be slow.
+ // So, right now, we will have Infinity for those BigDecimal number.
+ // TODO: Support BigDecimal.
+ json.mapPartitions(iter => {
+ // When there is a key appearing multiple times (a duplicate key),
+ // the ObjectMapper will take the last value associated with this duplicate key.
+ // For example: for {"key": 1, "key":2}, we will get "key"->2.
+ val mapper = new ObjectMapper()
+ iter.map(record => mapper.readValue(record, classOf[Object]))
+ }).map(scalafy).map(_.asInstanceOf[Map[String, Any]])
+ }
+
+ protected def toLong(value: Any): Long = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Long]
+ case value: java.lang.Long => value.asInstanceOf[Long]
+ }
+ }
+
+ protected def toDouble(value: Any): Double = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Double]
+ case value: java.lang.Long => value.asInstanceOf[Long].asInstanceOf[Double]
+ case value: java.lang.Double => value.asInstanceOf[Double]
+ }
+ }
+
+ protected def toDecimal(value: Any): BigDecimal = {
+ value match {
+ case value: java.lang.Integer => BigDecimal(value)
+ case value: java.lang.Long => BigDecimal(value)
+ case value: java.math.BigInteger => BigDecimal(value)
+ case value: java.lang.Double => BigDecimal(value)
+ case value: java.math.BigDecimal => BigDecimal(value)
+ }
+ }
+
+ protected def enforceCorrectType(value: Any, desiredType: DataType): Any ={
+ if (value == null) {
+ null
+ } else {
+ desiredType match {
+ case ArrayType(elementType) =>
+ value.asInstanceOf[Seq[Any]].map(enforceCorrectType(_, elementType))
+ case StringType => value.toString
+ case IntegerType => value.asInstanceOf[IntegerType.JvmType]
--- End diff --
Relatedly, can we add a few tests for type promotion? @aarondav has been running into some issues with the current implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520869
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
--- End diff --
typo
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13892881
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -99,6 +97,35 @@ class SQLContext(@transient val sparkContext: SparkContext)
new SchemaRDD(this, parquet.ParquetRelation(path))
/**
+ * Loads a JSON file (one object per line), returning the result as a [[SchemaRDD]].
+ *
+ * @group userf
+ */
+ def jsonFile(path: String): SchemaRDD = jsonFile(path, 1.0)
+
+ /**
+ * :: Experimental ::
+ */
--- End diff --
here too, although with sampling
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520512
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
+ Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
+ // Boolean is only wider than Void
+ val booleanPrecedence = Seq(NullType, BooleanType)
+ val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ protected def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ // TODO: Returns the union of fields1 and fields2?
+ case (StructType(fields1), StructType(fields2))
+ if (fields1 == fields2) => StructType(fields1)
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ protected def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // We comment out the following line in the development to catch bugs.
+ // We need to enable this line in future to handle
+ // unexpected data type.
+ // case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ protected def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When a value of a key is an object, we
+ * only use a placeholder for a struct type (StructType(Nil)) instead of getting
+ * all fields of this struct because a field does not appear in this JSON object
+ * can appear in other JSON objects.
+ */
+ protected def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
+ * We do not use Jackson's scala module at here because
+ * DefaultScalaModule in jackson-module-scala will make
+ * the parsing very slow.
+ */
+ protected def scalafy(obj: Any): Any = obj match {
+ case map: java.util.Map[String, Object] =>
+ // .map(identity) is used as a workaround of non-serializable Map
+ // generated by .mapValues.
+ // This issue is documented at https://issues.scala-lang.org/browse/SI-7005
+ map.toMap.mapValues(scalafy).map(identity)
+ case list: java.util.List[Object] =>
+ list.toList.map(scalafy)
+ case atom => atom
+ }
+
+ protected def parseJson(json: RDD[String]): RDD[Map[String, Any]] = {
+ // According to [Jackson-72: https://jira.codehaus.org/browse/JACKSON-72],
+ // ObjectMapper will not return BigDecimal when
+ // "DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" is disabled
+ // (see NumberDeserializer.deserialize for the logic).
+ // But, we do not want to enable this feature because it will use BigDecimal
+ // for every float number, which will be slow.
+ // So, right now, we will have Infinity for those BigDecimal number.
+ // TODO: Support BigDecimal.
+ json.mapPartitions(iter => {
+ // When there is a key appearing multiple times (a duplicate key),
+ // the ObjectMapper will take the last value associated with this duplicate key.
+ // For example: for {"key": 1, "key":2}, we will get "key"->2.
+ val mapper = new ObjectMapper()
+ iter.map(record => mapper.readValue(record, classOf[Object]))
+ }).map(scalafy).map(_.asInstanceOf[Map[String, Any]])
+ }
+
+ protected def toLong(value: Any): Long = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Long]
+ case value: java.lang.Long => value.asInstanceOf[Long]
+ }
+ }
+
+ protected def toDouble(value: Any): Double = {
+ value match {
+ case value: java.lang.Integer => value.asInstanceOf[Int].asInstanceOf[Double]
+ case value: java.lang.Long => value.asInstanceOf[Long].asInstanceOf[Double]
+ case value: java.lang.Double => value.asInstanceOf[Double]
+ }
+ }
+
+ protected def toDecimal(value: Any): BigDecimal = {
+ value match {
+ case value: java.lang.Integer => BigDecimal(value)
+ case value: java.lang.Long => BigDecimal(value)
+ case value: java.math.BigInteger => BigDecimal(value)
+ case value: java.lang.Double => BigDecimal(value)
+ case value: java.math.BigDecimal => BigDecimal(value)
+ }
+ }
+
+ protected def enforceCorrectType(value: Any, desiredType: DataType): Any ={
+ if (value == null) {
+ null
+ } else {
+ desiredType match {
+ case ArrayType(elementType) =>
+ value.asInstanceOf[Seq[Any]].map(enforceCorrectType(_, elementType))
+ case StringType => value.toString
+ case IntegerType => value.asInstanceOf[IntegerType.JvmType]
--- End diff --
These sorts of casts are only going to do what you want if the type of value isn't `Any`
```scala
scala> 1L.asInstanceOf[Int]
res0: Int = 1
scala> (1L: Any).asInstanceOf[Int]
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45382130
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46389597
Is that the basic strategy we are going to use with AlphaComponents -- merging new APIs at both the minor and maintenance levels? I don't know that I have any objection to that, but I don't recall any discussion directly on point, and this is the first such addition that has been made to branch-1.0 while I was paying attention.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46436035
Hmmm, that doesn't precisely match my recollection or understanding. Certainly we discussed that alpha components aren't required to maintain a stable API, but I don't recall an explicit decision that changes to alpha components would routinely be merged back into maintenance releases. I could be mistaken, and merging new alpha API into maintenance branches may be the right strategy, but this did take me a little by surprise.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13829803
--- Diff: examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQL.java ---
@@ -56,6 +57,8 @@ public static void main(String[] args) throws Exception {
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
JavaSQLContext sqlCtx = new JavaSQLContext(ctx);
+
--- End diff --
extra line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46047262
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828272
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala ---
@@ -108,19 +118,18 @@ trait HiveTypeCoercion {
*
* Additionally, all types when UNION-ed with strings will be promoted to strings.
* Other string conversions are handled by PromoteStrings.
+ *
+ * Widening types might result in loss of precision in the following cases:
+ * - IntegerType to FloatType
+ * - LongType to FloatType
+ * - LongType to DoubleType
*/
object WidenTypes extends Rule[LogicalPlan] {
- // See https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types.
- // The conversion for integral and floating point types have a linear widening hierarchy:
- val numericPrecedence =
- Seq(NullType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, DecimalType)
- // Boolean is only wider than Void
- val booleanPrecedence = Seq(NullType, BooleanType)
- val allPromotions: Seq[Seq[DataType]] = numericPrecedence :: booleanPrecedence :: Nil
def findTightestCommonType(t1: DataType, t2: DataType): Option[DataType] = {
// Try and find a promotion rule that contains both types in question.
- val applicableConversion = allPromotions.find(p => p.contains(t1) && p.contains(t2))
+ val applicableConversion = HiveTypeCoercion.allPromotions.find(p => p.contains(t1) && p
--- End diff --
In general I'd prefer to break at a higher syntactic level. (i.e., right after the && or even put the whole command on the next line with only the `val applicableConversion =` on the line above).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46352124
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13827652
--- Diff: docs/sql-programming-guide.md ---
@@ -310,37 +325,190 @@ parquetFile = sqlCtx.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerAsTable("parquetFile");
teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
-
+teenNames = teenagers.map(lambda p: "Name: " + p.name)
+for teenName in teenNames.collect():
+ print teenName
{% endhighlight %}
</div>
</div>
-## Writing Language-Integrated Relational Queries
+## JSON Datasets
+<div class="codetabs">
+
+<div data-lang="scala" markdown="1">
+Spark SQL supports querying JSON datasets. To query a JSON dataset, a SchemaRDD needs to be created for this JSON dataset. There are two ways to create a SchemaRDD for a JSON dataset:
--- End diff --
How about:
Spark SQL can automatically infer the schema of a JSON dataset and load it as a SchemaRDD. This conversion can be done using one of two methods:
- `jsonFile` - loads data from a directory of JSON files where each line of the files is a JSON object.
- `jsonRdd` - loads data from an existing RDD where each element of the RDD is a string containing a JSON object.
We should probably also have a brief section on the rules of schema inference as they are non-obvious.
A future TODO: There is no reason that we need the json objects to be one per line right? We can know when an object has not ended and could continue reading past a line break, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45638494
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15624/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828482
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/package.scala ---
@@ -17,8 +17,55 @@
package org.apache.spark.sql.catalyst
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.types.{ArrayType, DataType, StructField, StructType}
+
/**
* A a collection of common abstractions for query plans as well as
* a base logical plan representation.
*/
-package object plans
+package object plans {
+
+ def generateSchemaTreeString(schema: Seq[Attribute]): String = {
--- End diff --
is this only used in QueryPlan? If so, maybe is should just live there as a protected method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45814468
http://yhuai.github.io/spark-sql-core/api/#org.apache.spark.sql.json.JsonTable goes through how a user should use JsonTable.
Right now, a user can optionally sample the dataset for the process of schema inferring. It is possible that the schema based on the sampled dataset has missing fields and wrong data types. In this case, after the execution of first user query, the user should call adjustSchema to see if the schema has been updated. After the first execution, the JsonTable should have the complete schema.
Also, it will be good that we can provide facilities to let users refine the inferred schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45571771
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15588/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45571770
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520450
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
--- End diff --
Good idea, I like that better than Option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13839554
--- Diff: docs/sql-programming-guide.md ---
@@ -64,8 +64,8 @@ descendants. To create a basic SQLContext, all you need is a SparkContext.
val sc: SparkContext // An existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
-// Importing the SQL context gives access to all the public SQL functions and implicit conversions.
-import sqlContext._
+// createSchemaRDD is used to implicitly convert a RDD to a SchemaRDD.
--- End diff --
Instead of "a RDD" it should say "an RDD". There are several instances of that in this pr, would be good to check them out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45642612
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46047235
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46222354
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45383280
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46047249
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520549
--- Diff: project/SparkBuild.scala ---
@@ -486,9 +486,23 @@ object SparkBuild extends Build {
def sqlCoreSettings = sharedSettings ++ Seq(
name := "spark-sql",
libraryDependencies ++= Seq(
- "com.twitter" % "parquet-column" % parquetVersion,
- "com.twitter" % "parquet-hadoop" % parquetVersion
- )
+ "com.twitter" % "parquet-column" % parquetVersion,
+ "com.twitter" % "parquet-hadoop" % parquetVersion,
+ "com.fasterxml.jackson.core" % "jackson-core" % "2.3.2"
+ ),
+ initialCommands in console :=
+ """
+ |import org.apache.spark.sql.catalyst.analysis._
--- End diff --
If the console is primarily for developers then I find it pretty useful to have all the sorts of things I'd want for debugging in scope. This is how hive/console is already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46363827
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46101589
Also another note for @marmbrus as well: I'd really try to minimize the use of top-level SQLContext methods being called without `context.` in front of them due to `import SQLContext._`. This is confusing for newcomers to Scala and makes the code harder to translate across languages. You can *maybe* use it for `sql()` and `hql()` but even then I'd consider moving the implicit conversions somewhere else in a future version of the API (e.g. `import SQLContext._` to give you those). Let's discuss that more later, for now though it would be nice to update the examples to avoid these for `jsonFile`, `registerRDDAsTable`, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46250551
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13893257
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala ---
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.Logging
+
+private[sql] object JsonRDD extends Logging {
+
+ private[sql] def inferSchema(
+ json: RDD[String],
+ samplingRatio: Double = 1.0): LogicalPlan = {
+ require(samplingRatio > 0)
--- End diff --
add a more meaningful exception message, i.e.
```
require(samplingRatio > 0, s"samplingRatio ($samplingRatio) should be greater than 0")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13892709
--- Diff: sql/core/pom.xml ---
@@ -54,6 +61,11 @@
<version>${parquet.version}</version>
</dependency>
<dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>2.3.2</version>
--- End diff --
@pwendell I think in general sub project pom files don't specify dependency versions. Can you verify?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518312
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
--- End diff --
BTW sometimes sample is too expensive because sample isn't by blocks. It still requires reading all the data from disk. You might want to consider sampling by blocks here (or we can improve Spark's sample operator to do that).@mengxr any thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46372514
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46249486
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46363845
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46264604
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15837/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-45383113
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46360655
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15853/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13892635
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala ---
@@ -123,4 +125,53 @@ abstract class QueryPlan[PlanType <: TreeNode[PlanType]] extends TreeNode[PlanTy
case other => Nil
}.toSeq
}
+
+ protected def generateSchemaTreeString(schema: Seq[Attribute]): String = {
+ val builder = new StringBuilder
+ builder.append("root\n")
+ val prefix = " |"
+ schema.foreach { attribute =>
+ val name = attribute.name
+ val dataType = attribute.dataType
+ dataType match {
+ case fields: StructType =>
+ builder.append(s"$prefix-- $name: $StructType\n")
+ generateSchemaTreeString(fields, s"$prefix |", builder)
+ case ArrayType(fields: StructType) =>
+ builder.append(s"$prefix-- $name: $ArrayType[$StructType]\n")
+ generateSchemaTreeString(fields, s"$prefix |", builder)
+ case ArrayType(elementType: DataType) =>
+ builder.append(s"$prefix-- $name: $ArrayType[$elementType]\n")
+ case _ => builder.append(s"$prefix-- $name: $dataType\n")
+ }
+ }
+
+ builder.toString()
+ }
+
+ protected def generateSchemaTreeString(
+ schema: StructType,
+ prefix: String,
+ builder: StringBuilder): StringBuilder = {
+ schema.fields.foreach {
+ case StructField(name, fields: StructType, _) =>
+ builder.append(s"$prefix-- $name: $StructType\n")
+ generateSchemaTreeString(fields, s"$prefix |", builder)
+ case StructField(name, ArrayType(fields: StructType), _) =>
+ builder.append(s"$prefix-- $name: $ArrayType[$StructType]\n")
+ generateSchemaTreeString(fields, s"$prefix |", builder)
+ case StructField(name, ArrayType(elementType: DataType), _) =>
+ builder.append(s"$prefix-- $name: $ArrayType[$elementType]\n")
+ case StructField(name, fieldType: DataType, _) =>
+ builder.append(s"$prefix-- $name: $fieldType\n")
+ }
+
+ builder
+ }
+
+ /** Returns the output schema in the tree format. */
+ def schemaTreeString: String = generateSchemaTreeString(output)
--- End diff --
maybe just schemaString
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46046107
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13893273
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala ---
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.Logging
+
+private[sql] object JsonRDD extends Logging {
+
+ private[sql] def inferSchema(
+ json: RDD[String],
+ samplingRatio: Double = 1.0): LogicalPlan = {
+ require(samplingRatio > 0)
+ val schemaData = if (samplingRatio > 0.99) json else json.sample(false, samplingRatio, 1)
+
--- End diff --
probably no need to have a blank line for each statement ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13827732
--- Diff: docs/sql-programming-guide.md ---
@@ -310,37 +325,190 @@ parquetFile = sqlCtx.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerAsTable("parquetFile");
teenagers = sqlCtx.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
-
+teenNames = teenagers.map(lambda p: "Name: " + p.name)
+for teenName in teenNames.collect():
+ print teenName
{% endhighlight %}
</div>
</div>
-## Writing Language-Integrated Relational Queries
+## JSON Datasets
+<div class="codetabs">
+
+<div data-lang="scala" markdown="1">
+Spark SQL supports querying JSON datasets. To query a JSON dataset, a SchemaRDD needs to be created for this JSON dataset. There are two ways to create a SchemaRDD for a JSON dataset:
-**Language-Integrated queries are currently only supported in Scala.**
+1. Creating the SchemaRDD from text files that store one JSON object per line.
+2. Creating the SchemaRDD from a RDD of strings (`RDD[String]`) that stores one JSON object.
-Spark SQL also supports a domain specific language for writing queries. Once again,
-using the data from the above examples:
+The schema of a JSON dataset is automatically inferred when the SchemaRDD is created.
{% highlight scala %}
-val sqlContext = new org.apache.spark.sql.SQLContext(sc)
-import sqlContext._
-val people: RDD[Person] = ... // An RDD of case class objects, from the first example.
+// sc is an existing SparkContext.
+val sqlCtx = new org.apache.spark.sql.SQLContext(sc)
+
+// A JSON dataset is pointed by path.
+// The path can be either a single text file or a directory storing text files.
+val path = "examples/src/main/resources/people.json"
+// Create a SchemaRDD from the file(s) pointed by path
+val people = sqlCtx.jsonFile(path)
+
+// Because the schema of a JSON dataset is automatically inferred, to write queries,
+// it is better to take a look at what is the schema.
+people.printSchema()
+// The schema of people is ...
+// root
+// |-- age: IntegerType
+// |-- name: StringType
+
+// Register this SchemaRDD as a table.
+people.registerAsTable("people")
-// The following is the same as 'SELECT name FROM people WHERE age >= 10 AND age <= 19'
-val teenagers = people.where('age >= 10).where('age <= 19).select('name)
+// SQL statements can be run by using the sql methods provided by sqlCtx.
+val teenagers = sqlCtx.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
+
+// The results of SQL queries are SchemaRDDs and support all the normal RDD operations.
+// The columns of a row in the result can be accessed by ordinal.
+teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
--- End diff --
This is probably overkill as we have shown this multiple times in this guide.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13830433
--- Diff: docs/sql-programming-guide.md ---
@@ -62,10 +62,10 @@ descendants. To create a basic SQLContext, all you need is a SparkContext.
{% highlight scala %}
val sc: SparkContext // An existing SparkContext.
-val sqlContext = new org.apache.spark.sql.SQLContext(sc)
+val sqlCtx = new org.apache.spark.sql.SQLContext(sc)
--- End diff --
i see. I was trying to make those names consistent in the doc for these three languages. I will use sqlContext.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13523010
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
--- End diff --
Yup this shouldn't block on that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13518328
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
--- End diff --
isn't this already defined in catalyst type system?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13601819
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala ---
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.catalyst.expressions.{ExprId, AttributeReference, Attribute}
+import org.apache.spark.sql.catalyst.plans.generateSchemaTreeString
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.catalyst.util._
+
+class JsonSuite extends QueryTest {
+ import TestJsonData._
+ TestJsonData
+
+ /**
+ * Since attribute references are given globally unique ids during analysis,
+ * we must normalize them to check if two different queries are identical.
+ */
+ protected def normalizeExprIds(attributes: Seq[Attribute]) = {
+ val minId = attributes.map(_.exprId.id).min
+ attributes.map {
+ case a: AttributeReference =>
+ AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId))
+ }
+ }
+
+ protected def checkSchema(expected: Seq[Attribute], actual: Seq[Attribute]): Unit = {
+ val normalizedExpected = normalizeExprIds(expected).toSeq
+ val normalizedActual = normalizeExprIds(actual).toSeq
+ if (normalizedExpected != normalizedActual) {
+ fail(
+ s"""
+ |=== FAIL: Schemas do not match ===
+ |${sideBySide(
+ s"== Expected Schema ==\n" +
+ generateSchemaTreeString(normalizedExpected),
+ s"== Actual Schema ==\n" +
+ generateSchemaTreeString(normalizedActual)).mkString("\n")}
+ """.stripMargin)
+ }
+ }
+
+ test("Primitive field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(primitiveFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("bigInteger", DecimalType, true)() ::
+ AttributeReference("boolean", BooleanType, true)() ::
+ AttributeReference("double", DoubleType, true)() ::
+ AttributeReference("integer", IntegerType, true)() ::
+ AttributeReference("long", LongType, true)() ::
+ AttributeReference("null", StringType, true)() ::
+ AttributeReference("string", StringType, true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ checkAnswer(
+ sql("select * from jsonTable"),
+ (BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157E308,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.") :: Nil
+ )
+ }
+
+ test("Complex field and type inferring") {
+ val jsonSchemaRDD = jsonRDD(complexFieldAndType)
+
+ val expectedSchema =
+ AttributeReference("arrayOfArray1", ArrayType(ArrayType(StringType)), true)() ::
+ AttributeReference("arrayOfArray2", ArrayType(ArrayType(DoubleType)), true)() ::
+ AttributeReference("arrayOfBigInteger", ArrayType(DecimalType), true)() ::
+ AttributeReference("arrayOfBoolean", ArrayType(BooleanType), true)() ::
+ AttributeReference("arrayOfDouble", ArrayType(DoubleType), true)() ::
+ AttributeReference("arrayOfInteger", ArrayType(IntegerType), true)() ::
+ AttributeReference("arrayOfLong", ArrayType(LongType), true)() ::
+ AttributeReference("arrayOfNull", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfString", ArrayType(StringType), true)() ::
+ AttributeReference("arrayOfStruct", ArrayType(
+ StructType(StructField("field1", BooleanType, true) ::
+ StructField("field2", StringType, true) :: Nil)), true)() ::
+ AttributeReference("struct", StructType(
+ StructField("field1", BooleanType, true) ::
+ StructField("field2", DecimalType, true) :: Nil), true)() ::
+ AttributeReference("structWithArrayFields", StructType(
+ StructField("field1", ArrayType(IntegerType), true) ::
+ StructField("field2", ArrayType(StringType), true) :: Nil), true)() :: Nil
+
+ checkSchema(expectedSchema, jsonSchemaRDD.logicalPlan.output)
+
+ jsonSchemaRDD.registerAsTable("jsonTable")
+
+ // Access elements of a primitive array.
+ checkAnswer(
+ sql("select arrayOfString[0], arrayOfString[1], arrayOfString[2] from jsonTable"),
+ ("str1", "str2", null) :: Nil
+ )
+
+ // Access an array of null values.
+ checkAnswer(
+ sql("select arrayOfNull from jsonTable"),
+ Seq(Seq(null, null, null, null)) :: Nil
+ )
+
+ // Access elements of a BigInteger array (we use DecimalType internally).
+ checkAnswer(
+ sql("select arrayOfBigInteger[0], arrayOfBigInteger[1], arrayOfBigInteger[2] from jsonTable"),
+ (BigDecimal("922337203685477580700"), BigDecimal("-922337203685477580800"), null) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray1[0], arrayOfArray1[1] from jsonTable"),
+ (Seq("1", "2", "3"), Seq("str1", "str2")) :: Nil
+ )
+
+ // Access elements of an array of arrays.
+ checkAnswer(
+ sql("select arrayOfArray2[0], arrayOfArray2[1] from jsonTable"),
+ (Seq(1.0, 2.0, 3.0), Seq(1.1, 2.1, 3.1)) :: Nil
+ )
+
+ // Access elements of an array inside a filed with the type of ArrayType(ArrayType).
+ checkAnswer(
+ sql("select arrayOfArray1[1][1], arrayOfArray2[1][1] from jsonTable"),
+ ("str2", 2.1) :: Nil
+ )
+
+ // Access elements of an array of structs.
+ checkAnswer(
+ sql("select arrayOfStruct[0], arrayOfStruct[1], arrayOfStruct[2] from jsonTable"),
+ (true :: "str1" :: Nil, false :: null :: Nil, null) :: Nil
+ )
+
+ /*
+ // Right now, "field1" and "field2" are treated as aliases. We should fix it.
+ // TODO: Re-enable the following test.
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ (true, "str1") :: Nil
+ )
+ */
+
+ /*
+ // Right now, the analyzer cannot resolve arrayOfStruct.field1 and arrayOfStruct.field2.
--- End diff --
https://issues.apache.org/jira/browse/SPARK-2096
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13520424
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.SchemaRDD
+import org.apache.spark.sql.Logging
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, GetField}
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.types.StructField
+import org.apache.spark.sql.catalyst.types.StructType
+import org.apache.spark.sql.catalyst.types.ArrayType
+import org.apache.spark.sql.catalyst.expressions.GetField
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.execution.SparkLogicalPlan
+import org.apache.spark.sql.catalyst.expressions.Alias
+
+sealed trait SchemaResolutionMode
+
+case object EAGER_SCHEMA_RESOLUTION extends SchemaResolutionMode
+case class EAGER_SCHEMA_RESOLUTION_WITH_SAMPLING(val fraction: Double) extends SchemaResolutionMode
+case object LAZY_SCHEMA_RESOLUTION extends SchemaResolutionMode
+
+/**
+ * :: Experimental ::
+ * Converts a JSON file to a SparkSQL logical query plan. This implementation is only designed to
+ * work on JSON files that have mostly uniform schema. The conversion suffers from the following
+ * limitation:
+ * - The data is optionally sampled to determine all of the possible fields. Any fields that do
+ * not appear in this sample will not be included in the final output.
+ */
+@Experimental
+object JsonTable extends Serializable with Logging {
+ def inferSchema(
+ json: RDD[String], sampleSchema: Option[Double] = None): LogicalPlan = {
+ val schemaData = sampleSchema.map(json.sample(false, _, 1)).getOrElse(json)
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ // Finally, we replace all NullType to StringType. We do not need to take care
+ // StructType because all fields with a StructType are represented by a placeholder
+ // StructType(Nil).
+ dataType match {
+ case NullType => (fieldName, StringType)
+ case ArrayType(NullType) => (fieldName, ArrayType(StringType))
+ case other => (fieldName, other)
+ }
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq.sortBy {
+ case StructField(name, _, _) => name
+ }
+
+ StructType(topLevelFields ++ structFields)
+ }
+
+ val schema = makeStruct(resolved.keySet.toSeq, Nil)
+
+ SparkLogicalPlan(
+ ExistingRdd(
+ asAttributes(schema),
+ parseJson(json).map(asRow(_, schema))))
+ }
+
+ // numericPrecedence and booleanPrecedence are from WidenTypes.
+ // A widening conversion of a value with IntegerType and LongType to FloatType,
+ // or of a value with LongType to DoubleType, may result in loss of precision
+ // (some of the least significant bits of the value).
+ val numericPrecedence =
--- End diff --
Only in a Hive based Rule. If this is the exact same ordering though we might consider pulling it out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2060][SQL] Querying JSON Datasets with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/999#issuecomment-46383321
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/999#discussion_r13828804
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala ---
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.json
+
+import scala.collection.JavaConversions._
+import scala.math.BigDecimal
+
+import com.fasterxml.jackson.databind.ObjectMapper
+
+import org.apache.spark.annotation.{DeveloperApi, Experimental}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.sql.Logging
+
+@Experimental
+private[sql] object JsonRDD extends Logging {
+
+ @DeveloperApi
+ private[sql] def inferSchema(
+ json: RDD[String],
+ samplingRatio: Double = 1.0): LogicalPlan = {
+ require(samplingRatio > 0)
+ val schemaData = if (samplingRatio > 0.99) json else json.sample(false, samplingRatio, 1)
+
+ val allKeys = parseJson(schemaData).map(getAllKeysWithValueTypes).reduce(_ ++ _)
+
+ val baseSchema = createSchema(allKeys)
+
+ createLogicalPlan(json, baseSchema)
+ }
+
+ private def createLogicalPlan(
+ json: RDD[String],
+ baseSchema: StructType): LogicalPlan = {
+ val schema = nullTypeToStringType(baseSchema)
+
+ SparkLogicalPlan(ExistingRdd(asAttributes(schema), parseJson(json).map(asRow(_, schema))))
+ }
+
+ private def createSchema(allKeys: Set[(String, DataType)]): StructType = {
+ // Resolve type conflicts
+ val resolved = allKeys.groupBy {
+ case (key, dataType) => key
+ }.map {
+ // Now, keys and types are organized in the format of
+ // key -> Set(type1, type2, ...).
+ case (key, typeSet) => {
+ val fieldName = key.substring(1, key.length - 1).split("`.`").toSeq
+ val dataType = typeSet.map {
+ case (_, dataType) => dataType
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ (fieldName, dataType)
+ }
+ }
+
+ def makeStruct(values: Seq[Seq[String]], prefix: Seq[String]): StructType = {
+ val (topLevel, structLike) = values.partition(_.size == 1)
+ val topLevelFields = topLevel.filter {
+ name => resolved.get(prefix ++ name).get match {
+ case ArrayType(StructType(Nil)) => false
+ case ArrayType(_) => true
+ case struct: StructType => false
+ case _ => true
+ }
+ }.map {
+ a => StructField(a.head, resolved.get(prefix ++ a).get, nullable = true)
+ }
+
+ val structFields: Seq[StructField] = structLike.groupBy(_(0)).map {
+ case (name, fields) => {
+ val nestedFields = fields.map(_.tail)
+ val structType = makeStruct(nestedFields, prefix :+ name)
+ val dataType = resolved.get(prefix :+ name).get
+ dataType match {
+ case array: ArrayType => Some(StructField(name, ArrayType(structType), nullable = true))
+ case struct: StructType => Some(StructField(name, structType, nullable = true))
+ // dataType is StringType means that we have resolved type conflicts involving
+ // primitive types and complex types. So, the type of name has been relaxed to
+ // StringType. Also, this field should have already been put in topLevelFields.
+ case StringType => None
+ }
+ }
+ }.flatMap(field => field).toSeq
+
+ StructType(
+ (topLevelFields ++ structFields).sortBy {
+ case StructField(name, _, _) => name
+ })
+ }
+
+ makeStruct(resolved.keySet.toSeq, Nil)
+ }
+
+ /**
+ * Returns the most general data type for two given data types.
+ */
+ private[json] def getCompatibleType(t1: DataType, t2: DataType): DataType = {
+ // Try and find a promotion rule that contains both types in question.
+ val applicableConversion = HiveTypeCoercion.allPromotions.find(p => p.contains(t1) && p
+ .contains(t2))
+
+ // If found return the widest common type, otherwise None
+ val returnType = applicableConversion.map(_.filter(t => t == t1 || t == t2).last)
+
+ if (returnType.isDefined) {
+ returnType.get
+ } else {
+ // t1 or t2 is a StructType, ArrayType, or an unexpected type.
+ (t1, t2) match {
+ case (other: DataType, NullType) => other
+ case (NullType, other: DataType) => other
+ case (StructType(fields1), StructType(fields2)) => {
+ val newFields = (fields1 ++ fields2).groupBy(field => field.name).map {
+ case (name, fieldTypes) => {
+ val dataType = fieldTypes.map(field => field.dataType).reduce(
+ (type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+ StructField(name, dataType, true)
+ }
+ }
+ StructType(newFields.toSeq.sortBy {
+ case StructField(name, _, _) => name
+ })
+ }
+ case (ArrayType(elementType1), ArrayType(elementType2)) =>
+ ArrayType(getCompatibleType(elementType1, elementType2))
+ // TODO: We should use JsonObjectStringType to mark that values of field will be
+ // strings and every string is a Json object.
+ case (_, _) => StringType
+ }
+ }
+ }
+
+ private def getPrimitiveType(value: Any): DataType = {
+ value match {
+ case value: java.lang.String => StringType
+ case value: java.lang.Integer => IntegerType
+ case value: java.lang.Long => LongType
+ // Since we do not have a data type backed by BigInteger,
+ // when we see a Java BigInteger, we use DecimalType.
+ case value: java.math.BigInteger => DecimalType
+ case value: java.lang.Double => DoubleType
+ case value: java.math.BigDecimal => DecimalType
+ case value: java.lang.Boolean => BooleanType
+ case null => NullType
+ // Unexpected data type.
+ case _ => StringType
+ }
+ }
+
+ /**
+ * Returns the element type of an JSON array. We go through all elements of this array
+ * to detect any possible type conflict. We use [[getCompatibleType]] to resolve
+ * type conflicts. Right now, when the element of an array is another array, we
+ * treat the element as String.
+ */
+ private def getTypeOfArray(l: Seq[Any]): ArrayType = {
+ val elements = l.flatMap(v => Option(v))
+ if (elements.isEmpty) {
+ // If this JSON array is empty, we use NullType as a placeholder.
+ // If this array is not empty in other JSON objects, we can resolve
+ // the type after we have passed through all JSON objects.
+ ArrayType(NullType)
+ } else {
+ val elementType = elements.map {
+ e => e match {
+ case map: Map[_, _] => StructType(Nil)
+ // We have an array of arrays. If those element arrays do not have the same
+ // element types, we will return ArrayType[StringType].
+ case seq: Seq[_] => getTypeOfArray(seq)
+ case value => getPrimitiveType(value)
+ }
+ }.reduce((type1: DataType, type2: DataType) => getCompatibleType(type1, type2))
+
+ ArrayType(elementType)
+ }
+ }
+
+ /**
+ * Figures out all key names and data types of values from a parsed JSON object
+ * (in the format of Map[Stirng, Any]). When the value of a key is an JSON object, we
+ * only use a placeholder (StructType(Nil)) to mark that it should be a struct
+ * instead of getting all fields of this struct because a field does not appear
+ * in this JSON object can appear in other JSON objects.
+ */
+ private def getAllKeysWithValueTypes(m: Map[String, Any]): Set[(String, DataType)] = {
+ m.map{
+ // Quote the key with backticks to handle cases which have dots
+ // in the field name.
+ case (key, dataType) => (s"`$key`", dataType)
+ }.flatMap {
+ case (key: String, struct: Map[String, Any]) => {
+ // The value associted with the key is an JSON object.
+ getAllKeysWithValueTypes(struct).map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } ++ Set((key, StructType(Nil)))
+ }
+ case (key: String, array: List[Any]) => {
+ // The value associted with the key is an array.
+ getTypeOfArray(array) match {
+ case ArrayType(StructType(Nil)) => {
+ // The elements of this arrays are structs.
+ array.asInstanceOf[List[Map[String, Any]]].flatMap {
+ element => getAllKeysWithValueTypes(element)
+ }.map {
+ case (k, dataType) => (s"$key.$k", dataType)
+ } :+ (key, ArrayType(StructType(Nil)))
+ }
+ case ArrayType(elementType) => (key, ArrayType(elementType)) :: Nil
+ }
+ }
+ case (key: String, value) => (key, getPrimitiveType(value)) :: Nil
+ }.toSet
+ }
+
+ /**
+ * Converts a Java Map/List to a Scala Map/List.
--- End diff --
Is there a reason we can't just `import scala.collection.JavaConversions._`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---