You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:21:36 UTC
[jira] [Updated] (SPARK-11202) Unsupported dataType
[ https://issues.apache.org/jira/browse/SPARK-11202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-11202:
---------------------------------
Labels: bulk-closed (was: )
> Unsupported dataType
> --------------------
>
> Key: SPARK-11202
> URL: https://issues.apache.org/jira/browse/SPARK-11202
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: whc
> Priority: Major
> Labels: bulk-closed
>
> I read data from oracle and save as parquet ,then get the following error:
> java.lang.IllegalArgumentException: Unsupported dataType: {"type":"struct","fields":[{"name":"DOMAIN_NAME","type":"string","nullable":true,"metadata":{"name":"DOMAIN_NAME"}},{"name":"DOMAIN_ID","type":"decimal(0,-127)","nullable":true,"metadata":{"name":"DOMAIN_ID"}}]}, [1.1] failure: `TimestampType' expected but `{' found
> {"type":"struct","fields":[{"name":"DOMAIN_NAME","type":"string","nullable":true,"metadata":{"name":"DOMAIN_NAME"}},{"name":"DOMAIN_ID","type":"decimal(0,-127)","nullable":true,"metadata":{"name":"DOMAIN_ID"}}]}
> ^
> at org.apache.spark.sql.types.DataType$CaseClassStringParser$.apply(DataType.scala:245)
> at org.apache.spark.sql.types.DataType$.fromCaseClassString(DataType.scala:102)
> at org.apache.spark.sql.execution.datasources.parquet.ParquetTypesConverter$$anonfun$3.apply(ParquetTypesConverter.scala:62)
> at org.apache.spark.sql.execution.datasources.parquet.ParquetTypesConverter$$anonfun$3.apply(ParquetTypesConverter.scala:62)
> at scala.util.Try.getOrElse(Try.scala:77)
> at org.apache.spark.sql.execution.datasources.parquet.ParquetTypesConverter$.convertFromString(ParquetTypesConverter.scala:62)
> at org.apache.spark.sql.execution.datasources.parquet.RowWriteSupport.init(ParquetTableSupport.scala:51)
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:288)
> at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
> at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetRelation.scala:94)
> at org.apache.spark.sql.execution.datasources.parquet.ParquetRelation$$anon$3.newInstance(ParquetRelation.scala:272)
> at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:234)
> at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
> at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> I checked the type but do not have Timestamp or Date type in oracle
> my oracle table like this:
> create table DW_DOMAIN
> (
> domain_id NUMBER,
> cityid NUMBER,
> domain_type NUMBER,
> domain_name VARCHAR2(80)
> )
> and my code like this:
> Map<String, String> options = new HashMap<String, String>();
> options.put("url", url);
> options.put("driver", driver);
> options.put("user", user);
> options.put("password", password);
> options.put("dbtable", "(select DOMAIN_NAME,DOMAIN_ID from dw_domain ) t");
> DataFrame df = this.sqlContext.read().format("jdbc").options(options )
> .load();
> df.write().mode(SaveMode.Append)
> .parquet("hdfs://cluster1:8020/database/count_domain/");
> if add "to_char(DOMAIN_ID)",that can get correct result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org