You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by tridib <tr...@live.com> on 2014/10/21 01:34:11 UTC
spark sql: timestamp in json - fails
Hello Experts,
After repeated attempt I am unable to run query on map json date string. I
tried two approaches:
*** Approach 1 *** created a Bean class with timespamp field. When I try to
run it I get scala.MatchError: class java.sql.Timestamp (of class
java.lang.Class). Here is the code:
import java.sql.Timestamp;
public class ComplexClaim {
private Timestamp timestamp;
public Timestamp getTimestamp() {
return timestamp;
}
public void setTimestamp(Timestamp timestamp) {
this.timestamp = timestamp;
}
}
JavaSparkContext ctx = getCtx(sc);
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
ComplexClaim.class);
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
*** Approach 2 ***
Created a StructType to map the date field. I got scala.MatchError:
TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$).
here is the code:
public StructType createStructType() {
List<StructField> fields = new ArrayList<StructField>();
fields.add(DataType.createStructField("timestamp",
DataType.TimestampType, false));
return DataType.createStructType(fields);
}
public void testJsonStruct(SparkContext sc) {
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
}
Input file has a single record:
{"timestamp":"2014-10-10T01:01:01"}
Thanks
Tridib
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark sql: timestamp in json - fails
Posted by "Wang, Daoyuan" <da...@intel.com>.
Yes, SPARK-3853 just got merged 11 days ago. It should be OK in 1.2.0. And for the first approach, It would be ok after SPARK-4003 is merged.
-----Original Message-----
From: tridib [mailto:tridib.samanta@live.com]
Sent: Tuesday, October 21, 2014 11:09 AM
To: user@spark.incubator.apache.org
Subject: RE: spark sql: timestamp in json - fails
Spark 1.1.0
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864p16888.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark sql: timestamp in json - fails
Posted by tridib <tr...@live.com>.
Spark 1.1.0
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864p16888.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark sql: timestamp in json - fails
Posted by "Wang, Daoyuan" <da...@intel.com>.
The exception of second approach, has been resolved by SPARK-3853.
Thanks,
Daoyuan
-----Original Message-----
From: Wang, Daoyuan [mailto:daoyuan.wang@intel.com]
Sent: Tuesday, October 21, 2014 11:06 AM
To: tridib; user@spark.incubator.apache.org
Subject: RE: spark sql: timestamp in json - fails
That's weird, I think we have that Pattern match in enforceCorrectType. What version of spark are you using?
Thanks,
Daoyuan
-----Original Message-----
From: tridib [mailto:tridib.samanta@live.com]
Sent: Tuesday, October 21, 2014 11:03 AM
To: user@spark.incubator.apache.org
Subject: Re: spark sql: timestamp in json - fails
Stack trace for my second case:
2014-10-20 23:00:36,903 ERROR [Executor task launch worker-0] executor.Executor (Logging.scala:logError(96)) - Exception in task 0.0 in stage 0.0 (TID 0)
scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
at
org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:348)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
at scala.Option.map(Option.scala:145)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
at
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at
scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
at org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-10-20 23:00:36,933 WARN [Result resolver thread-1] scheduler.TaskSetManager (Logging.scala:logWarning(71)) - Lost task 0.0 in stage 0.0 (TID 0, localhost): scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:348)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
scala.Option.map(Option.scala:145)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$class.foreach(Iterator.scala:727)
scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
scala.collection.AbstractIterator.to(Iterator.scala:1157)
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2014-10-20 23:00:36,935 ERROR [Result resolver thread-1] scheduler.TaskSetManager (Logging.scala:logError(75)) - Task 0 in stage 0.0 failed 1 times; aborting job
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864p16884.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark sql: timestamp in json - fails
Posted by "Wang, Daoyuan" <da...@intel.com>.
That's weird, I think we have that Pattern match in enforceCorrectType. What version of spark are you using?
Thanks,
Daoyuan
-----Original Message-----
From: tridib [mailto:tridib.samanta@live.com]
Sent: Tuesday, October 21, 2014 11:03 AM
To: user@spark.incubator.apache.org
Subject: Re: spark sql: timestamp in json - fails
Stack trace for my second case:
2014-10-20 23:00:36,903 ERROR [Executor task launch worker-0] executor.Executor (Logging.scala:logError(96)) - Exception in task 0.0 in stage 0.0 (TID 0)
scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
at
org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:348)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
at scala.Option.map(Option.scala:145)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
at
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at
scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
at org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-10-20 23:00:36,933 WARN [Result resolver thread-1] scheduler.TaskSetManager (Logging.scala:logWarning(71)) - Lost task 0.0 in stage 0.0 (TID 0, localhost): scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:348)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
scala.Option.map(Option.scala:145)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$class.foreach(Iterator.scala:727)
scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
scala.collection.AbstractIterator.to(Iterator.scala:1157)
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2014-10-20 23:00:36,935 ERROR [Result resolver thread-1] scheduler.TaskSetManager (Logging.scala:logError(75)) - Task 0 in stage 0.0 failed 1 times; aborting job
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864p16884.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark sql: timestamp in json - fails
Posted by tridib <tr...@live.com>.
Stack trace for my second case:
2014-10-20 23:00:36,903 ERROR [Executor task launch worker-0]
executor.Executor (Logging.scala:logError(96)) - Exception in task 0.0 in
stage 0.0 (TID 0)
scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
at
org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:348)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
at scala.Option.map(Option.scala:145)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
at
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at
scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
at org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-10-20 23:00:36,933 WARN [Result resolver thread-1]
scheduler.TaskSetManager (Logging.scala:logWarning(71)) - Lost task 0.0 in
stage 0.0 (TID 0, localhost): scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:348)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
scala.Option.map(Option.scala:145)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$class.foreach(Iterator.scala:727)
scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
scala.collection.AbstractIterator.to(Iterator.scala:1157)
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
org.apache.spark.rdd.RDD$$anonfun$16.apply(RDD.scala:774)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2014-10-20 23:00:36,935 ERROR [Result resolver thread-1]
scheduler.TaskSetManager (Logging.scala:logError(75)) - Task 0 in stage 0.0
failed 1 times; aborting job
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864p16884.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark sql: timestamp in json - fails
Posted by "Wang, Daoyuan" <da...@intel.com>.
Seems I made a mistake…
From: Wang, Daoyuan
Sent: Tuesday, October 21, 2014 10:35 AM
To: 'Yin Huai'
Cc: Michael Armbrust; tridib; user@spark.incubator.apache.org
Subject: RE: spark sql: timestamp in json - fails
I got that, it is in JsonRDD.java of `typeOfPrimitiveValues`. I’ll fix that together.
Thanks,
Daoyuan
From: Yin Huai [mailto:huaiyin.thu@gmail.com]
Sent: Tuesday, October 21, 2014 10:13 AM
To: Wang, Daoyuan
Cc: Michael Armbrust; tridib; user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Subject: Re: spark sql: timestamp in json - fails
Seems the second approach does not go through applySchema. So, I was wondering if there is an issue related to our JSON apis in Java.
On Mon, Oct 20, 2014 at 10:04 PM, Wang, Daoyuan <da...@intel.com>> wrote:
I think this has something to do with my recent work at https://issues.apache.org/jira/browse/SPARK-4003
You can check PR https://github.com/apache/spark/pull/2850 .
Thanks,
Daoyuan
From: Yin Huai [mailto:huaiyin.thu@gmail.com<ma...@gmail.com>]
Sent: Tuesday, October 21, 2014 10:00 AM
To: Michael Armbrust
Cc: tridib; user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Subject: Re: spark sql: timestamp in json - fails
Hi Tridib,
For the second approach, can you attach the complete stack trace?
Thanks,
Yin
On Mon, Oct 20, 2014 at 8:24 PM, Michael Armbrust <mi...@databricks.com>> wrote:
I think you are running into a bug that will be fixed by this PR: https://github.com/apache/spark/pull/2850
On Mon, Oct 20, 2014 at 4:34 PM, tridib <tr...@live.com>> wrote:
Hello Experts,
After repeated attempt I am unable to run query on map json date string. I
tried two approaches:
*** Approach 1 *** created a Bean class with timespamp field. When I try to
run it I get scala.MatchError: class java.sql.Timestamp (of class
java.lang.Class). Here is the code:
import java.sql.Timestamp;
public class ComplexClaim {
private Timestamp timestamp;
public Timestamp getTimestamp() {
return timestamp;
}
public void setTimestamp(Timestamp timestamp) {
this.timestamp = timestamp;
}
}
JavaSparkContext ctx = getCtx(sc);
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
ComplexClaim.class);
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
*** Approach 2 ***
Created a StructType to map the date field. I got scala.MatchError:
TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$).
here is the code:
public StructType createStructType() {
List<StructField> fields = new ArrayList<StructField>();
fields.add(DataType.createStructField("timestamp",
DataType.TimestampType, false));
return DataType.createStructType(fields);
}
public void testJsonStruct(SparkContext sc) {
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
}
Input file has a single record:
{"timestamp":"2014-10-10T01:01:01"}
Thanks
Tridib
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
RE: spark sql: timestamp in json - fails
Posted by "Wang, Daoyuan" <da...@intel.com>.
I got that, it is in JsonRDD.java of `typeOfPrimitiveValues`. I’ll fix that together.
Thanks,
Daoyuan
From: Yin Huai [mailto:huaiyin.thu@gmail.com]
Sent: Tuesday, October 21, 2014 10:13 AM
To: Wang, Daoyuan
Cc: Michael Armbrust; tridib; user@spark.incubator.apache.org
Subject: Re: spark sql: timestamp in json - fails
Seems the second approach does not go through applySchema. So, I was wondering if there is an issue related to our JSON apis in Java.
On Mon, Oct 20, 2014 at 10:04 PM, Wang, Daoyuan <da...@intel.com>> wrote:
I think this has something to do with my recent work at https://issues.apache.org/jira/browse/SPARK-4003
You can check PR https://github.com/apache/spark/pull/2850 .
Thanks,
Daoyuan
From: Yin Huai [mailto:huaiyin.thu@gmail.com<ma...@gmail.com>]
Sent: Tuesday, October 21, 2014 10:00 AM
To: Michael Armbrust
Cc: tridib; user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Subject: Re: spark sql: timestamp in json - fails
Hi Tridib,
For the second approach, can you attach the complete stack trace?
Thanks,
Yin
On Mon, Oct 20, 2014 at 8:24 PM, Michael Armbrust <mi...@databricks.com>> wrote:
I think you are running into a bug that will be fixed by this PR: https://github.com/apache/spark/pull/2850
On Mon, Oct 20, 2014 at 4:34 PM, tridib <tr...@live.com>> wrote:
Hello Experts,
After repeated attempt I am unable to run query on map json date string. I
tried two approaches:
*** Approach 1 *** created a Bean class with timespamp field. When I try to
run it I get scala.MatchError: class java.sql.Timestamp (of class
java.lang.Class). Here is the code:
import java.sql.Timestamp;
public class ComplexClaim {
private Timestamp timestamp;
public Timestamp getTimestamp() {
return timestamp;
}
public void setTimestamp(Timestamp timestamp) {
this.timestamp = timestamp;
}
}
JavaSparkContext ctx = getCtx(sc);
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
ComplexClaim.class);
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
*** Approach 2 ***
Created a StructType to map the date field. I got scala.MatchError:
TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$).
here is the code:
public StructType createStructType() {
List<StructField> fields = new ArrayList<StructField>();
fields.add(DataType.createStructField("timestamp",
DataType.TimestampType, false));
return DataType.createStructType(fields);
}
public void testJsonStruct(SparkContext sc) {
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
}
Input file has a single record:
{"timestamp":"2014-10-10T01:01:01"}
Thanks
Tridib
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
Re: spark sql: timestamp in json - fails
Posted by Yin Huai <hu...@gmail.com>.
Seems the second approach does not go through applySchema. So, I was
wondering if there is an issue related to our JSON apis in Java.
On Mon, Oct 20, 2014 at 10:04 PM, Wang, Daoyuan <da...@intel.com>
wrote:
> I think this has something to do with my recent work at
> https://issues.apache.org/jira/browse/SPARK-4003
>
> You can check PR https://github.com/apache/spark/pull/2850 .
>
>
>
> Thanks,
>
> Daoyuan
>
>
>
> *From:* Yin Huai [mailto:huaiyin.thu@gmail.com]
> *Sent:* Tuesday, October 21, 2014 10:00 AM
> *To:* Michael Armbrust
> *Cc:* tridib; user@spark.incubator.apache.org
> *Subject:* Re: spark sql: timestamp in json - fails
>
>
>
> Hi Tridib,
>
>
>
> For the second approach, can you attach the complete stack trace?
>
>
>
> Thanks,
>
>
>
> Yin
>
>
>
> On Mon, Oct 20, 2014 at 8:24 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
> I think you are running into a bug that will be fixed by this PR:
> https://github.com/apache/spark/pull/2850
>
>
>
> On Mon, Oct 20, 2014 at 4:34 PM, tridib <tr...@live.com> wrote:
>
> Hello Experts,
> After repeated attempt I am unable to run query on map json date string. I
> tried two approaches:
>
> *** Approach 1 *** created a Bean class with timespamp field. When I try to
> run it I get scala.MatchError: class java.sql.Timestamp (of class
> java.lang.Class). Here is the code:
> import java.sql.Timestamp;
>
> public class ComplexClaim {
> private Timestamp timestamp;
>
> public Timestamp getTimestamp() {
> return timestamp;
> }
>
> public void setTimestamp(Timestamp timestamp) {
> this.timestamp = timestamp;
> }
> }
>
> JavaSparkContext ctx = getCtx(sc);
> JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
> String path = "/hdd/spark/test.json";
> JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
> ComplexClaim.class);
> sqlCtx.registerRDDAsTable(test, "test");
> execSql(sqlCtx, "select * from test", 1);
>
>
>
> *** Approach 2 ***
> Created a StructType to map the date field. I got scala.MatchError:
> TimestampType (of class
> org.apache.spark.sql.catalyst.types.TimestampType$).
> here is the code:
>
> public StructType createStructType() {
> List<StructField> fields = new ArrayList<StructField>();
> fields.add(DataType.createStructField("timestamp",
> DataType.TimestampType, false));
> return DataType.createStructType(fields);
> }
>
> public void testJsonStruct(SparkContext sc) {
> JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
> String path = "/hdd/spark/test.json";
> JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
> sqlCtx.registerRDDAsTable(test, "test");
> execSql(sqlCtx, "select * from test", 1);
> }
>
> Input file has a single record:
> {"timestamp":"2014-10-10T01:01:01"}
>
>
> Thanks
> Tridib
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
>
>
RE: spark sql: timestamp in json - fails
Posted by "Wang, Daoyuan" <da...@intel.com>.
I think this has something to do with my recent work at https://issues.apache.org/jira/browse/SPARK-4003
You can check PR https://github.com/apache/spark/pull/2850 .
Thanks,
Daoyuan
From: Yin Huai [mailto:huaiyin.thu@gmail.com]
Sent: Tuesday, October 21, 2014 10:00 AM
To: Michael Armbrust
Cc: tridib; user@spark.incubator.apache.org
Subject: Re: spark sql: timestamp in json - fails
Hi Tridib,
For the second approach, can you attach the complete stack trace?
Thanks,
Yin
On Mon, Oct 20, 2014 at 8:24 PM, Michael Armbrust <mi...@databricks.com>> wrote:
I think you are running into a bug that will be fixed by this PR: https://github.com/apache/spark/pull/2850
On Mon, Oct 20, 2014 at 4:34 PM, tridib <tr...@live.com>> wrote:
Hello Experts,
After repeated attempt I am unable to run query on map json date string. I
tried two approaches:
*** Approach 1 *** created a Bean class with timespamp field. When I try to
run it I get scala.MatchError: class java.sql.Timestamp (of class
java.lang.Class). Here is the code:
import java.sql.Timestamp;
public class ComplexClaim {
private Timestamp timestamp;
public Timestamp getTimestamp() {
return timestamp;
}
public void setTimestamp(Timestamp timestamp) {
this.timestamp = timestamp;
}
}
JavaSparkContext ctx = getCtx(sc);
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
ComplexClaim.class);
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
*** Approach 2 ***
Created a StructType to map the date field. I got scala.MatchError:
TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$).
here is the code:
public StructType createStructType() {
List<StructField> fields = new ArrayList<StructField>();
fields.add(DataType.createStructField("timestamp",
DataType.TimestampType, false));
return DataType.createStructType(fields);
}
public void testJsonStruct(SparkContext sc) {
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
}
Input file has a single record:
{"timestamp":"2014-10-10T01:01:01"}
Thanks
Tridib
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
Re: spark sql: timestamp in json - fails
Posted by Yin Huai <hu...@gmail.com>.
Hi Tridib,
For the second approach, can you attach the complete stack trace?
Thanks,
Yin
On Mon, Oct 20, 2014 at 8:24 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> I think you are running into a bug that will be fixed by this PR:
> https://github.com/apache/spark/pull/2850
>
> On Mon, Oct 20, 2014 at 4:34 PM, tridib <tr...@live.com> wrote:
>
>> Hello Experts,
>> After repeated attempt I am unable to run query on map json date string. I
>> tried two approaches:
>>
>> *** Approach 1 *** created a Bean class with timespamp field. When I try
>> to
>> run it I get scala.MatchError: class java.sql.Timestamp (of class
>> java.lang.Class). Here is the code:
>> import java.sql.Timestamp;
>>
>> public class ComplexClaim {
>> private Timestamp timestamp;
>>
>> public Timestamp getTimestamp() {
>> return timestamp;
>> }
>>
>> public void setTimestamp(Timestamp timestamp) {
>> this.timestamp = timestamp;
>> }
>> }
>>
>> JavaSparkContext ctx = getCtx(sc);
>> JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
>> String path = "/hdd/spark/test.json";
>> JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
>> ComplexClaim.class);
>> sqlCtx.registerRDDAsTable(test, "test");
>> execSql(sqlCtx, "select * from test", 1);
>>
>>
>>
>> *** Approach 2 ***
>> Created a StructType to map the date field. I got scala.MatchError:
>> TimestampType (of class
>> org.apache.spark.sql.catalyst.types.TimestampType$).
>> here is the code:
>>
>> public StructType createStructType() {
>> List<StructField> fields = new ArrayList<StructField>();
>> fields.add(DataType.createStructField("timestamp",
>> DataType.TimestampType, false));
>> return DataType.createStructType(fields);
>> }
>>
>> public void testJsonStruct(SparkContext sc) {
>> JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
>> String path = "/hdd/spark/test.json";
>> JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
>> sqlCtx.registerRDDAsTable(test, "test");
>> execSql(sqlCtx, "select * from test", 1);
>> }
>>
>> Input file has a single record:
>> {"timestamp":"2014-10-10T01:01:01"}
>>
>>
>> Thanks
>> Tridib
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: spark sql: timestamp in json - fails
Posted by Michael Armbrust <mi...@databricks.com>.
I think you are running into a bug that will be fixed by this PR:
https://github.com/apache/spark/pull/2850
On Mon, Oct 20, 2014 at 4:34 PM, tridib <tr...@live.com> wrote:
> Hello Experts,
> After repeated attempt I am unable to run query on map json date string. I
> tried two approaches:
>
> *** Approach 1 *** created a Bean class with timespamp field. When I try to
> run it I get scala.MatchError: class java.sql.Timestamp (of class
> java.lang.Class). Here is the code:
> import java.sql.Timestamp;
>
> public class ComplexClaim {
> private Timestamp timestamp;
>
> public Timestamp getTimestamp() {
> return timestamp;
> }
>
> public void setTimestamp(Timestamp timestamp) {
> this.timestamp = timestamp;
> }
> }
>
> JavaSparkContext ctx = getCtx(sc);
> JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
> String path = "/hdd/spark/test.json";
> JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
> ComplexClaim.class);
> sqlCtx.registerRDDAsTable(test, "test");
> execSql(sqlCtx, "select * from test", 1);
>
>
>
> *** Approach 2 ***
> Created a StructType to map the date field. I got scala.MatchError:
> TimestampType (of class
> org.apache.spark.sql.catalyst.types.TimestampType$).
> here is the code:
>
> public StructType createStructType() {
> List<StructField> fields = new ArrayList<StructField>();
> fields.add(DataType.createStructField("timestamp",
> DataType.TimestampType, false));
> return DataType.createStructType(fields);
> }
>
> public void testJsonStruct(SparkContext sc) {
> JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
> String path = "/hdd/spark/test.json";
> JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
> sqlCtx.registerRDDAsTable(test, "test");
> execSql(sqlCtx, "select * from test", 1);
> }
>
> Input file has a single record:
> {"timestamp":"2014-10-10T01:01:01"}
>
>
> Thanks
> Tridib
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>