You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Tobias Pfeiffer <tg...@preferred.jp> on 2015/01/16 08:25:59 UTC
MatchError in JsonRDD.toLong
Hi,
I am experiencing a weird error that suddenly popped up in my unit tests. I
have a couple of HDFS files in JSON format and my test is basically
creating a JsonRDD and then issuing a very simple SQL query over it. This
used to work fine, but now suddenly I get:
15:58:49.039 [Executor task launch worker-1] ERROR executor.Executor -
Exception in task 1.0 in stage 29.0 (TID 117)
scala.MatchError: 14452800566866169008 (of class java.math.BigInteger)
at org.apache.spark.sql.json.JsonRDD$.toLong(JsonRDD.scala:282)
at org.apache.spark.sql.json.JsonRDD$.enforceCorrectType(JsonRDD.scala:353)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1$$anonfun$apply$12.apply(JsonRDD.scala:381)
at scala.Option.map(Option.scala:145)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:380)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$asRow$1.apply(JsonRDD.scala:365)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.sql.json.JsonRDD$.org$apache$spark$sql$json$JsonRDD$$asRow(JsonRDD.scala:365)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
at
org.apache.spark.sql.json.JsonRDD$$anonfun$jsonStringToRow$1.apply(JsonRDD.scala:38)
...
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
The stack trace contains none of my classes, so it's a bit hard to track
down where this starts.
The code of JsonRDD.toLong is in fact
private def toLong(value: Any): Long = {
value match {
case value: java.lang.Integer => value.asInstanceOf[Int].toLong
case value: java.lang.Long => value.asInstanceOf[Long]
}
}
so if value is a BigInteger, toLong doesn't work. Now I'm wondering where
this comes from (I haven't touched this component in a while, nor upgraded
Spark etc.), but in particular I would like to know how to work around this.
Thanks
Tobias
RE: MatchError in JsonRDD.toLong
Posted by "Wang, Daoyuan" <da...@intel.com>.
Yes, actually that is what I mean exactly. And maybe you missed my last response, you can use the API:
jsonRDD(json:RDD[String], schema:StructType)
to clearly clarify your schema. For numbers bigger than Long, we can use DecimalType.
Thanks,
Daoyuan
From: Tobias Pfeiffer [mailto:tgp@preferred.jp]
Sent: Tuesday, January 20, 2015 9:26 AM
To: Wang, Daoyuan
Cc: user
Subject: Re: MatchError in JsonRDD.toLong
Hi,
On Fri, Jan 16, 2015 at 6:14 PM, Wang, Daoyuan <da...@intel.com>> wrote:
The second parameter of jsonRDD is the sampling ratio when we infer schema.
OK, I was aware of this, but I guess I understand the problem now. My sampling ratio is so low that I only see the Long values of data items and infer it's a Long. When I meet the data that's actually longer than Long, I get the error I posted; basically it's the same situation as when specifying a wrong schema manually.
So is there any way around this other than increasing the sample ratio to discover also the very BigDecimal-sized numbers?
Thanks
Tobias
Re: MatchError in JsonRDD.toLong
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Fri, Jan 16, 2015 at 6:14 PM, Wang, Daoyuan <da...@intel.com>
wrote:
> The second parameter of jsonRDD is the sampling ratio when we infer schema.
>
OK, I was aware of this, but I guess I understand the problem now. My
sampling ratio is so low that I only see the Long values of data items and
infer it's a Long. When I meet the data that's actually longer than Long, I
get the error I posted; basically it's the same situation as when
specifying a wrong schema manually.
So is there any way around this other than increasing the sample ratio to
discover also the very BigDecimal-sized numbers?
Thanks
Tobias
RE: MatchError in JsonRDD.toLong
Posted by "Wang, Daoyuan" <da...@intel.com>.
And you can use jsonRDD(json:RDD[String], schema:StructType) to clearly clarify your schema. For numbers later than Long, we can use DecimalType.
Thanks,
Daoyuan
From: Wang, Daoyuan [mailto:daoyuan.wang@intel.com]
Sent: Friday, January 16, 2015 5:14 PM
To: Tobias Pfeiffer
Cc: user
Subject: RE: MatchError in JsonRDD.toLong
The second parameter of jsonRDD is the sampling ratio when we infer schema.
Thanks,
Daoyuan
From: Tobias Pfeiffer [mailto:tgp@preferred.jp]
Sent: Friday, January 16, 2015 5:11 PM
To: Wang, Daoyuan
Cc: user
Subject: Re: MatchError in JsonRDD.toLong
Hi,
On Fri, Jan 16, 2015 at 5:55 PM, Wang, Daoyuan <da...@intel.com>> wrote:
Can you provide how you create the JsonRDD?
This should be reproducible in the Spark shell:
---------------------------------------------------------
import org.apache.spark.sql._
val sqlc = new SparkContext(sc)
val rdd = sc.parallelize("""{"Click":"nonclicked", "Impression":1, "DisplayURL":4401798909506983219, "AdId":21215341}""" ::
"""{"Click":"nonclicked", "Impression":1, "DisplayURL":14452800566866169008, "AdId":10587781}""" :: Nil)
// works fine
val json = sqlc.jsonRDD(rdd)
json.registerTempTable("test")
sqlc.sql("SELECT * FROM test").collect
// -> MatchError
val json2 = sqlc.jsonRDD(rdd, 0.1)
json2.registerTempTable("test2")
sqlc.sql("SELECT * FROM test2").collect
---------------------------------------------------------
I guess the issue in the latter case is that the column is inferred as Long when some rows actually are too big for Long...
Thanks
Tobias
RE: MatchError in JsonRDD.toLong
Posted by "Wang, Daoyuan" <da...@intel.com>.
The second parameter of jsonRDD is the sampling ratio when we infer schema.
Thanks,
Daoyuan
From: Tobias Pfeiffer [mailto:tgp@preferred.jp]
Sent: Friday, January 16, 2015 5:11 PM
To: Wang, Daoyuan
Cc: user
Subject: Re: MatchError in JsonRDD.toLong
Hi,
On Fri, Jan 16, 2015 at 5:55 PM, Wang, Daoyuan <da...@intel.com>> wrote:
Can you provide how you create the JsonRDD?
This should be reproducible in the Spark shell:
---------------------------------------------------------
import org.apache.spark.sql._
val sqlc = new SparkContext(sc)
val rdd = sc.parallelize("""{"Click":"nonclicked", "Impression":1, "DisplayURL":4401798909506983219, "AdId":21215341}""" ::
"""{"Click":"nonclicked", "Impression":1, "DisplayURL":14452800566866169008, "AdId":10587781}""" :: Nil)
// works fine
val json = sqlc.jsonRDD(rdd)
json.registerTempTable("test")
sqlc.sql("SELECT * FROM test").collect
// -> MatchError
val json2 = sqlc.jsonRDD(rdd, 0.1)
json2.registerTempTable("test2")
sqlc.sql("SELECT * FROM test2").collect
---------------------------------------------------------
I guess the issue in the latter case is that the column is inferred as Long when some rows actually are too big for Long...
Thanks
Tobias
Re: MatchError in JsonRDD.toLong
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Fri, Jan 16, 2015 at 5:55 PM, Wang, Daoyuan <da...@intel.com>
wrote:
>
> Can you provide how you create the JsonRDD?
>
This should be reproducible in the Spark shell:
---------------------------------------------------------
import org.apache.spark.sql._
val sqlc = new SparkContext(sc)
val rdd = sc.parallelize("""{"Click":"nonclicked", "Impression":1,
"DisplayURL":4401798909506983219, "AdId":21215341}""" ::
"""{"Click":"nonclicked", "Impression":1,
"DisplayURL":14452800566866169008, "AdId":10587781}""" :: Nil)
// works fine
val json = sqlc.jsonRDD(rdd)
json.registerTempTable("test")
sqlc.sql("SELECT * FROM test").collect
// -> MatchError
val json2 = sqlc.jsonRDD(rdd, 0.1)
json2.registerTempTable("test2")
sqlc.sql("SELECT * FROM test2").collect
---------------------------------------------------------
I guess the issue in the latter case is that the column is inferred as Long
when some rows actually are too big for Long...
Thanks
Tobias
RE: MatchError in JsonRDD.toLong
Posted by "Wang, Daoyuan" <da...@intel.com>.
Hi Tobias,
Can you provide how you create the JsonRDD?
Thanks,
Daoyuan
From: Tobias Pfeiffer [mailto:tgp@preferred.jp]
Sent: Friday, January 16, 2015 4:01 PM
To: user
Subject: Re: MatchError in JsonRDD.toLong
Hi again,
On Fri, Jan 16, 2015 at 4:25 PM, Tobias Pfeiffer <tg...@preferred.jp>> wrote:
Now I'm wondering where this comes from (I haven't touched this component in a while, nor upgraded Spark etc.) [...]
So the reason that the error is showing up now is that suddenly data from a different dataset is showing up in my test dataset... don't ask me... anyway, this different dataset contains data like
{"Click":"nonclicked", "Impression":1,
"DisplayURL":4401798909506983219, "AdId":21215341, ...}
{"Click":"nonclicked", "Impression":1,
"DisplayURL":14452800566866169008, "AdId":10587781, ...}
and the DisplayURL seems to be too long for Long, while it is still inferred as a Long column.
So, what to do about this? Is jsonRDD inherently incapable of handling those long numbers or is it just an issue in the schema inference and I should file a JIRA issue?
Thanks
Tobias
Re: MatchError in JsonRDD.toLong
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi again,
On Fri, Jan 16, 2015 at 4:25 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Now I'm wondering where this comes from (I haven't touched this component
> in a while, nor upgraded Spark etc.) [...]
>
So the reason that the error is showing up now is that suddenly data from a
different dataset is showing up in my test dataset... don't ask me...
anyway, this different dataset contains data like
{"Click":"nonclicked", "Impression":1,
"DisplayURL":4401798909506983219, "AdId":21215341, ...}
{"Click":"nonclicked", "Impression":1,
"DisplayURL":14452800566866169008, "AdId":10587781, ...}
and the DisplayURL seems to be too long for Long, while it is still
inferred as a Long column.
So, what to do about this? Is jsonRDD inherently incapable of handling
those long numbers or is it just an issue in the schema inference and I
should file a JIRA issue?
Thanks
Tobias