You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by critikaled <is...@gmail.com> on 2014/11/24 00:00:20 UTC
How to insert complex types like map> in
spark sql
Hi,
I am trying to insert particular set of data from rdd to a hive table I
have Map[String,Map[String,Int]] in scala which I want to insert into the
table of map<string,map<string,int>> I was able to create the table but
while inserting it says scala.MatchError:
MapType(StringType,MapType(StringType,IntegerType,true),true) (of class
org.apache.spark.sql.catalyst.types.MapType) can any one help me with this.
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map>
in spark sql
Posted by Cheng Lian <li...@gmail.com>.
Thanks Takuya! Will take a look into it later. And sorry for not being
able to review all the PRs in time recently (mostly because of rushing
Spark 1.2 release and Thanksgiving :) ).
On 11/27/14 1:35 AM, Takuya UESHIN wrote:
> Hi,
> I guess this is fixed by https://github.com/apache/spark/pull/3110
> which is not for complex type casting but makes inserting into hive
> table be able to handle complex types ignoring nullability.
>
> I also sent a pull-request (https://github.com/apache/spark/pull/3150)
> for complex type casting before.
> Please check it if #3110 doesn't work.
>
> Thanks.
>
>
> 2014-11-25 18:37 GMT+09:00 Cheng Lian <li...@gmail.com>:
>> Spark SQL supports complex types, but casting doesn't work for complex types
>> right now.
>>
>>
>> On 11/25/14 4:04 PM, critikaled wrote:
>>>
>>> https://github.com/apache/spark/blob/84d79ee9ec47465269f7b0a7971176da93c96f3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
>>>
>>> Doesn't look like spark sql support nested complex types right now
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19730.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map> in
spark sql
Posted by Takuya UESHIN <ue...@happy-camper.st>.
Hi,
I guess this is fixed by https://github.com/apache/spark/pull/3110
which is not for complex type casting but makes inserting into hive
table be able to handle complex types ignoring nullability.
I also sent a pull-request (https://github.com/apache/spark/pull/3150)
for complex type casting before.
Please check it if #3110 doesn't work.
Thanks.
2014-11-25 18:37 GMT+09:00 Cheng Lian <li...@gmail.com>:
> Spark SQL supports complex types, but casting doesn't work for complex types
> right now.
>
>
> On 11/25/14 4:04 PM, critikaled wrote:
>>
>>
>> https://github.com/apache/spark/blob/84d79ee9ec47465269f7b0a7971176da93c96f3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
>>
>> Doesn't look like spark sql support nested complex types right now
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19730.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
--
Takuya UESHIN
Tokyo, Japan
http://twitter.com/ueshin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map>
in spark sql
Posted by Cheng Lian <li...@gmail.com>.
Spark SQL supports complex types, but casting doesn't work for complex
types right now.
On 11/25/14 4:04 PM, critikaled wrote:
> https://github.com/apache/spark/blob/84d79ee9ec47465269f7b0a7971176da93c96f3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
>
> Doesn't look like spark sql support nested complex types right now
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19730.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map> in
spark sql
Posted by critikaled <is...@gmail.com>.
https://github.com/apache/spark/blob/84d79ee9ec47465269f7b0a7971176da93c96f3f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
Doesn't look like spark sql support nested complex types right now
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19730.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map> in
spark sql
Posted by critikaled <is...@gmail.com>.
Exactly that seems to be the problem will have to wait for the next release
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19734.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map> in
spark sql
Posted by critikaled <is...@gmail.com>.
Thanks for the reply Micheal here is the stack trace
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in
stage 0.0 failed 1 times, most recent failure: Lost task 3.0 in stage 0.0
(TID 3, localhost): scala.MatchError: MapType(StringType,StringType,true)
(of class org.apache.spark.sql.catalyst.types.MapType)
[info]
org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:247)
[info]
org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:247)
[info]
org.apache.spark.sql.catalyst.expressions.Cast.eval(Cast.scala:263)
[info]
org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:84)
[info]
org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:66)
[info]
org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:50)
[info] scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
[info] scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
[info]
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:149)
[info]
org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$1.apply(InsertIntoHiveTable.scala:158)
[info]
org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$1.apply(InsertIntoHiveTable.scala:158)
[info]
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
[info] org.apache.spark.scheduler.Task.run(Task.scala:54)
[info]
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
[info]
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[info]
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[info] java.lang.Thread.run(Thread.java:745)
[info] Driver stacktrace:
[info] at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
[info] at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[info] at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
[info] at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
[info] at scala.Option.foreach(Option.scala:236)
[info] at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603p19728.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to insert complex types like map> in
spark sql
Posted by Michael Armbrust <mi...@databricks.com>.
Can you give the full stack trace. You might be hitting:
https://issues.apache.org/jira/browse/SPARK-4293
On Sun, Nov 23, 2014 at 3:00 PM, critikaled <is...@gmail.com> wrote:
> Hi,
> I am trying to insert particular set of data from rdd to a hive table I
> have Map[String,Map[String,Int]] in scala which I want to insert into the
> table of map<string,map<string,int>> I was able to create the table but
> while inserting it says scala.MatchError:
> MapType(StringType,MapType(StringType,IntegerType,true),true) (of class
> org.apache.spark.sql.catalyst.types.MapType) can any one help me with this.
> Thanks in advance.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-insert-complex-types-like-map-string-map-string-int-in-spark-sql-tp19603.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>